[SPARK-2908] JsonRDD.nullTypeToStringType does not convert all NullType to StringType

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: SQL
• Target Version/s: 1.1.0
• Assignee: Yin Huai
• Reporter: Yin Huai
• Created: 07/Aug/14 20:38
• Updated: 08/Aug/14 18:10
• Resolved: 08/Aug/14 18:10

Description

Failed on

val schemaRDD = jsonRDD(sparkContext.parallelize(Seq("""{"record": [{"children": null}]}""")))

schemaRDD.schema
StructType(ArrayBuffer(StructField(record,ArrayType(StructType(ArrayBuffer(StructField(children,NullType,true))),false),true)))

You can see the children field is still NullType.

Issue Links

• is related to: SPARK-2695 Figure out a good way to handle NullType columns.
• links to: [Github] Pull Request #1840 (yhuai)