[HUDI-292] Consume more entries from kafka than specified sourceLimit.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: Utilities
• Labels: pull-request-available
• Assignee: leesf
• Reporter: leesf
• Created: 05/Oct/19 02:24
• Updated: 16/Oct/19 09:22
• Resolved: 11/Oct/19 15:32

Description

When CheckpointUtils#computeOffsetRanges for consuming kafka messges.

Given
topic = "test",
fromOffsets(partition -> offset pair) = (0 -> 0), (1 -> 0), (2 -> 0), (3 -> 0), (4 -> 0),
toOffsets = (0, 100), (1, 1000), (2, 1000), (3, 1000), (4, 1000),
numEvents = 1001.

The output of CheckpointUtils#computesOffsetRanges is

OffsetRange(topic: 'test', partition: 0, range: [0 -> 100])
OffsetRange(topic: 'test', partition: 1, range: [0 -> 226])
OffsetRange(topic: 'test', partition: 2, range: [0 -> 226])
OffsetRange(topic: 'test', partition: 3, range: [0 -> 226])
OffsetRange(topic: 'test', partition: 4, range: [0 -> 226])

Total count is 1004(100 + 266 * 4), more than 1001, and thus consume more entries from kafka than specified 1001.

CC vinoth

Issue Links

• links to: GitHub Pull Request #947