Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27494

Null keys/values don't work in Kafka source v2

    Details

      Description

      Right now Kafka source v2 doesn't support null keys or values.

      • When processing a null key, all of the following keys in the same partition will be null. This is a correctness bug.
      • When processing a null value, it will throw NPE.

      The workaround is setting sql conf "spark.sql.streaming.disabledV2MicroBatchReaders" to "org.apache.spark.sql.kafka010.KafkaSourceProvider" to use the v1 source.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                uncleGen Genmao Yu
                Reporter:
                zsxwing Shixiong Zhu
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: