[SPARK-32938] Spark can not cast long value from Kafka

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.0.0
• Component/s: Java API, SQL, Structured Streaming

Environment

Debian 10 (Buster), AMD64

Spark 3.0.0

Kafka 2.5.0

spark-sql-kafka-0-10_2.12

• Assignee: Unassigned
• Reporter: Matthias Seiler
• Created: 18/Sep/20 13:59
• Updated: 12/Dec/22 18:11
• Resolved: 20/Sep/20 01:50

Description

Spark seems to be unable to cast the key (or value) part from Kafka to a long value and throws

org.apache.spark.sql.AnalysisException: cannot resolve 'CAST(`key` AS BIGINT)' due to data type mismatch: cannot cast binary to bigint;;

 

See this repo for further investigation: https://github.com/maseiler/spark-kafka- casting-bug