[CRUNCH-609] KafkaSource could skip data with slow broker response

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Component/s: IO
• Assignee: Micah Whitacre
• Reporter: Micah Whitacre
• Created: 28/Jun/16 18:35
• Updated: 26/Feb/17 18:33
• Resolved: 13/Jul/16 13:26

Description

In the case of the consumer.poll timing out, the KafkaRecordReader could return with no values in the records iterator which then causes the KafkaRecordReader to exit prematurely[1]. To fix this we need to probably better track the start/end offset and that progress vs the null value.

Additionally if we wanted to be smarter we could do some "backoff" for the timeout or just have consumer specify a larger value.

[1] - https://github.com/apache/crunch/blob/360d72a4f887505e020fdb8f99c3ccb1800693f6/crunch-kafka/src/main/java/org/apache/crunch/kafka/inputformat/KafkaRecordReader.java#L122