[SPARK-12177] Update KafkaDStreams to new Kafka 0.10 Consumer API

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.0.0
• Component/s: DStreams
• Labels: consumer kafka
• Assignee: Cody Koeninger
• Reporter: Nikita Tarasenko
• Created: 07/Dec/15 11:26
• Updated: 22/Jul/16 16:39
• Resolved: 30/Jun/16 06:21

Description

Kafka 0.9 already released and it introduce new consumer API that not compatible with old one. So, I added new consumer api. I made separate classes in package org.apache.spark.streaming.kafka.v09 with changed API. I didn't remove old classes for more backward compatibility. User will not need to change his old spark applications when he uprgade to new Spark version.
Please rewiew my changes

Issue Links

• is blocked by: SPARK-13252 Bump up Kafka to 0.9.0.0
• is duplicated by: SPARK-15089 kafka-spark consumer with SSL problem
• relates to: SPARK-13252 Bump up Kafka to 0.9.0.0
• links to: [Github] Pull Request #11863 (koeninger)
• links to: [Github] Pull Request #13992 (tdas)
• links to: [Github] Pull Request #13996 (tdas)
• links to: [Github] Pull Request #13998 (koeninger)

Sub-Tasks

• Rename current streaming-kafka artifact to include kafka version: Resolved
• code cleanup of kafka-0-8 to match review feedback on 0-10: Resolved
• Docs for Kafka 0.10 consumer integration: Resolved
• unidoc workaround for multiple kafka versions: Resolved
• Kafka 0.10 Python support: In Progress