[SPARK-4068] NPE in jsonRDD schema inference

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: SQL
• Target Version/s: 1.2.0
• Assignee: Yin Huai
• Reporter: Michael Armbrust
• Created: 23/Oct/14 21:57
• Updated: 26/Oct/14 23:32
• Resolved: 26/Oct/14 23:32

Description

val jsonData = """{"data":[[null], [[["Test"]]]]}""" :: """{"other": ""}""" :: Nil
sqlContext.jsonRDD(sc.parallelize(jsonData))

org.apache.spark.SparkException: Job aborted due to stage failure: Task 13 in stage 5.0 failed 4 times, most recent failure: Lost task 13.3 in stage 5.0 (TID 347, ip-10-0-234-152.us-west-2.compute.internal): java.lang.NullPointerException: 
        org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$allKeysWithValueTypes$1.org$apache$spark$sql$json$JsonRDD$$anonfun$$buildKeyPathForInnerStructs$1(JsonRDD.scala:252)
        org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$allKeysWithValueTypes$1$$anonfun$org$apache$spark$sql$json$JsonRDD$$anonfun$$buildKeyPathForInnerStructs$1$3.apply(JsonRDD.scala:253)
        org.apache.spark.sql.json.JsonRDD$$anonfun$org$apache$spark$sql$json$JsonRDD$$allKeysWithValueTypes$1$$anonfun$org$apache$spark$sql$json$JsonRDD$$anonfun$$buildKeyPathForInnerStructs$1$3.apply(JsonRDD.scala:253)
        scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
        scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
...

Issue Links

• links to: [Github] Pull Request #2918 (yhuai)