[FLUME-2972] Handle offset migration in the new Kafka Channel

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.7.0
• Component/s: Channel
• Assignee: Grant Henke
• Reporter: Grant Henke
• Created: 15/Aug/16 21:47
• Updated: 01/Sep/16 01:50
• Resolved: 25/Aug/16 23:56

Description

Offsets tracking the position in Kafka consumers change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x.

FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off.

Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka.

Issue Links

• blocks: FLUME-2980 Automated concurrent Kafka offset migration test
• is related to: FLUME-2983 Handle offset migration in the new Kafka Source
• relates to: FLUME-2823 Flume-Kafka-Channel with new APIs
• links to: Review Board