[HIVE-20561] Use the position of the Kafka Consumer to track progress instead of Consumer Records offsets

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Slim Bouguerra Assign to me
• Reporter: Slim Bouguerra
• Created: 14/Sep/18 16:30
• Updated: 17/Nov/22 08:54
• Resolved: 18/Sep/18 17:56

Description

Kafka Partitions with transactional messages (post 0.11) will include commit or abort markers which indicate the result of a transaction. The markers are not returned to applications, yet have an offset in the log. Therefore the end of Stream position can be the offset of a control message.
This Patch change the way how we keep track of the consumer position by using

 consumer.position(topicP) 

as oppose to using the offset of the consumed messages.
Also I have done some refactoring to help code readability hopefully.

Issue Links

• relates to: HIVE-20377 Hive Kafka Storage Handler
• links to: RB Link