[SPARK-13252] Bump up Kafka to 0.9.0.0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.0.0
• Component/s: DStreams
• Labels: kafka
• Assignee: Unassigned
• Reporter: Mark Grover
• Created: 09/Feb/16 22:33
• Updated: 19/Apr/16 18:49
• Resolved: 19/Apr/16 18:49

Description

Apache Kafka release 0.9.0.0 came out some time and we should add support for it. This JIRA is related to SPARK-12177 which is related to adding support for new consumer API only available starting v0.9.0.0

However, we should upgrade Kafka to 0.9.0.0 regardless of when (and before) the support for the new consumer API gets added.

We also use some non-public APIs from Kafka which have changed in 0.9.0.0 release. So, this change should also take care of updating those usages.

Issue Links

• blocks: SPARK-12177 Update KafkaDStreams to new Kafka 0.10 Consumer API
• is duplicated by: SPARK-13251 Bump up Kafka to 0.9.0.0
• is related to: SPARK-12177 Update KafkaDStreams to new Kafka 0.10 Consumer API
• links to: [Github] Pull Request #11143 (markgrover)