[SPARK-18057] Update structured streaming kafka from 0.10.0.1 to 2.0.0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Target Version/s: 2.4.0
• Assignee: Ted Yu
• Reporter: Cody Koeninger
• Created: 21/Oct/16 18:21
• Updated: 05/Mar/19 21:46
• Resolved: 03/Aug/18 15:48

Description

There are a couple of relevant KIPs here, https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html

Issue Links

• blocks: SPARK-23539 Add support for Kafka headers in Structured Streaming
• causes: SPARK-25983 spark-sql-kafka-0-10 no longer works with Kafka 0.10.0
• is blocked by: KAFKA-2948 Kafka producer does not cope well with topic deletions
• is blocked by: KAFKA-4879 KafkaConsumer.position may hang forever when deleting a topic
• is blocked by: KAFKA-6608 Add TimeoutException to KafkaConsumer#position()
• is duplicated by: SPARK-25731 Spark Structured Streaming Support for Kafka 2.0
• is related to: SPARK-25954 Upgrade to Kafka 2.1.0
• relates to: KAFKA-4547 Consumer.position returns incorrect results for Kafka 0.10.1.0 client
• relates to: SPARK-18779 Messages being received only from one partition when using Spark Streaming integration for Kafka 0.10 with kafka client library at 0.10.1
• links to: [Github] Pull Request #21488 (tedyu)
• links to: [Github] Pull Request #21955 (srowen)
• links to: [Github] Pull Request #21995 (srowen)
• links to: [Github] Pull Request #22097 (zsxwing)