[FLINK-2386] Implement Kafka connector using the new Kafka Consumer API

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.1, 0.10.0
• Component/s: Connectors / Kafka
• Assignee: Robert Metzger
• Reporter: Robert Metzger
• Created: 21/Jul/15 15:09
• Updated: 27/Aug/15 11:35
• Resolved: 27/Aug/15 11:35

Description

Once Kafka has released its new consumer API, we should provide a connector for that version.
The release will probably be called 0.9 or 0.8.3.

The connector will be mostly compatible with Kafka 0.8.2.x, except for committing offsets to the broker (the new connector expects a coordinator to be available on Kafka). To work around that, we can provide a configuration option to commit offsets to zookeeper (managed by flink code).
For 0.9/0.8.3 it will be fully compatible.

It will not be compatible with 0.8.1 because of mismatching Kafka messages.