[SPARK-27494] Null keys/values don't work in Kafka source v2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1
• Fix Version/s: 2.4.3, 3.0.0
• Component/s: Structured Streaming
• Labels: correctness
• Assignee: Genmao Yu
• Reporter: Shixiong Zhu
• Created: 17/Apr/19 18:57
• Updated: 02/Mar/20 21:04
• Resolved: 26/Apr/19 06:29

Description

Right now Kafka source v2 doesn't support null keys or values.

• When processing a null key, all of the following keys in the same partition will be null. This is a correctness bug.
• When processing a null value, it will throw NPE.

The workaround is setting sql conf "spark.sql.streaming.disabledV2MicroBatchReaders" to "org.apache.spark.sql.kafka010.KafkaSourceProvider" to use the v1 source.

Issue Links

• links to: GitHub Pull Request #24441