who is responsible for offset maintenance?

120
June 19, 2017, at 02:31 AM

Here are the Kafka docs for public ConsumerRecords<K,V> poll(long timeout)

Fetch data for the topics or partitions specified using one of the subscribe/assign APIs. It is an error to not have subscribed to any topics or partitions before polling for data. On each poll, consumer will try to use the last consumed offset as the starting offset and fetch sequentially. The last consumed offset can be manually set through seek(TopicPartition, long) or automatically set as the last committed offset for the subscribed list of partitions

My question is who(Broker or consumer or zookeper) is responsible for maintaining the offset and where it is stored(memory or disc)? If consumer maintains it in memory, will consumer start reading it from beginning or consumer application need to persist in disc?

Answer 1

As the "Offsets and Consumer Position" section in the docs you referenced mentions, the offsets are stored by Kafka (the broker):

Kafka maintains a numerical offset for each record in a partition

Specifically, it stores them in an "internal" consumer offsets topic called "__consumer_offsets".

The "old consumer" api (deprecated in upcoming v0.11) allows you to chose to store offset in kafka or zookeeper.

Additionally, you are free to save offsets on the consumer side and always seek to those offsets at startup, if you so choose.

So, in summary, depending on your consumer api version and your preference, offsets can be stored on the broker or zookeeper and/or on the consumer side.

READ ALSO
Saving and Reading an ArrayList

Saving and Reading an ArrayList

Saving method(save list is an Arraylist)-

96
Show confirm dialog in Flux architecture

Show confirm dialog in Flux architecture

I'm trying to understand Flux architectureI'm using JavaFX and the FluxFX experimental framework

97
How to Compare a String with a Char

How to Compare a String with a Char

Guys how do i compare a String with a char ? heres my code :

110
What&#39;s wrong with my main method, and why is it throwing this exception? [on hold]

What's wrong with my main method, and why is it throwing this exception? [on hold]

I'm a new Java developer, writing my first real Java application, which is a calendar viewerHere's the code I wrote which is indicated as having errors:

108