[SPARK-29610] Keys with Null values are discarded when using to_json function

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.4
• Component/s: Build
• Assignee: Unassigned
• Reporter: Jonathan
• Created: 26/Oct/19 05:14
• Updated: 28/Oct/19 12:23
• Resolved: 28/Oct/19 12:23

Description

When calling to_json on a Struct if a key has Null as a value then the key is thrown away.

import pyspark
import pyspark.sql.functions as F

l = [("a", "foo"), ("b", None)]

df = spark.createDataFrame(l, ["id", "data"])
(
  df.select(F.struct("*").alias("payload"))
    .withColumn("payload", 
      F.to_json(F.col("payload"))
    ).select("payload")
    .show()
)

Produces the following output:

+--------------------+
|             payload|
+--------------------+
|{"id":"a","data":...|
|          {"id":"b"}|
+--------------------+

The `data` key in the second row has just been silently deleted.

Issue Links

• duplicates: SPARK-29444 Add configuration to support JacksonGenrator to keep fields with null values