[KAFKA-1977] Make logEndOffset available in the Zookeeper consumer - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Minor
Resolution: Auto Closed
Affects Version/s: None
Fix Version/s: None
Component/s: core
Labels:
None

Description

The requirement is to create a snapshot from the Kafka topic but NOT do continual reads after that point. For example you might be creating a backup of the data to a file.

In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps was to expose the high watermark, as maxEndOffset, from the FetchResponse object through to each MessageAndMetadata object in order to be aware when the consumer has reached the end of each partition.

The submitted patch achieves this by adding the maxEndOffset to the PartitionTopicInfo, which is updated when a new message arrives in the ConsumerFetcherThread and then exposed in MessageAndMetadata.

See here for discussion:
http://search-hadoop.com/m/4TaT4TpJy71