[SPARK-24189] Spark Strcutured Streaming not working with the Kafka Transactions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: bharath kumar avusherla
• Created: 05/May/18 05:05
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

Was trying to read kafka transactional topic using Spark Structured Streaming 2.3.0 with the  kafka option isolation-level = "read_committed", but spark reading the data immediately without waiting for the data in topic to be committed. In spark documentation it was mentioned as Structured Streaming supports Kafka version 0.10 or higher. I am using below command to test the scenario.

val df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "test-topic")
.option("isolation-level","read_committed")
.load()

Can you please let me know if the transactional read is supported in SPark 2.3.0 strcutured Streaming or am i missing anything.

 

Thank you.