[FLINK-4702] Kafka consumer must commit offsets asynchronously

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.2
• Fix Version/s: 1.1.3, 1.2.0
• Component/s: Connectors / Kafka
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 27/Sep/16 19:08
• Updated: 30/Sep/16 14:48
• Resolved: 30/Sep/16 14:48

Description

The offset commit calls to Kafka may occasionally take very long.
In that case, the notifyCheckpointComplete() method blocks for long and the KafkaConsumer cannot make progress and cannot perform checkpoints.

Kafka 0.9+ have methods to commit asynchronously.
We should use those and make sure no more than one commit is concurrently in progress, to that commit requests do not pile up.

Issue Links

• links to: GitHub Pull Request #2559
• links to: GitHub Pull Request #2574