[SPARK-25731] Spark Structured Streaming Support for Kafka 2.0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.2
• Component/s: Structured Streaming
• Labels: beginner features
• Assignee: Unassigned
• Reporter: Chandan
• Created: 15/Oct/18 08:54
• Updated: 12/Dec/22 18:10
• Resolved: 17/Oct/18 08:52

Description

https://github.com/apache/spark/tree/master/external

As far as I can see,
This doesn't have support for newly release kafka2.0,
support is available only till kafka-0-10.

If we use the 

"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0"

for kafka2.0, below is the error I get

11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Completed connection to node -1. Fetching API versions.
11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Initiating API versions fetch from node -1.
11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected
java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)

 

 

I might be wrong, but this is the best option I thought to open an issue. 

Issue Links

• duplicates: SPARK-18057 Update structured streaming kafka from 0.10.0.1 to 2.0.0