[SPARK-32834] from_avro is giving empty result

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: PySpark

Environment

Ubuntu 18

Spark 3.0

Kafka 2.0.0

• Assignee: Unassigned
• Reporter: Chaitanya
• Created: 09/Sep/20 17:52
• Updated: 15/Sep/20 00:40

Description

I am trying to read a Kafka topic with Spark readStream but getting problem while applying avro schema

 

Code:

df = spark\
  .readStream\
  .format("kafka")\
  .option("kafka.bootstrap.servers", "host:6667")\
  .option("subscribe", "utopic1")\
  .option("failOnDataLoss", "false")\
  .option("startingOffsets", "earliest")\
  .option("checkpointLocation", "/home/abc/wspace/spark_test/data/")\
  .load()
 
outputDF = df\
        .select(from_avro("value", jsonFormatSchema, options={"mode":"FASTFAIL"}).alias("user"))

outputDF.printSchema()

query = outputDF.writeStream.format("console").start()
time.sleep(10)

Input:

avro schema file: user.avsc

Kafka topic: {'favorite_color': 'Red', 'name': 'Alyssa'}

Expected Output:

It should print values. 

Actual Output:

+----+
|user|
+----+
|[,]|
+----+

Additional information:

• Searched in the internet and found that other peson faced same issue. https://stackoverflow.com/questions/59222774/spark-from-avro-function-returning-null-values
• I am able to print values to console if I cast to String using below line df.selectExpr("CAST(value AS STRING)")