[SPARK-19882] Pivot with null as the pivot value throws NPE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Andrew Ray
• Reporter: Hyukjin Kwon
• Created: 09/Mar/17 11:51
• Updated: 12/Dec/22 18:10
• Resolved: 17/Mar/17 08:44

Description

This seems a regression.

• Spark 1.6

Seq(Tuple1(None), Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()

prints

+----+----+---+
|   a|null|  1|
+----+----+---+
|null|   0|  0|
|   1|   0|  1|
+----+----+---+

• Current master

Seq(Tuple1(None), Tuple1(Some(1))).toDF("a").groupBy($"a").pivot("a").count().show()

prints

java.lang.NullPointerException was thrown.
java.lang.NullPointerException
  at org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:145)
  at org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst$$anonfun$4.apply(PivotFirst.scala:143)
  at scala.collection.immutable.List.map(List.scala:273)
  at org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.<init>(PivotFirst.scala:143)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot$$anonfun$apply$7$$anonfun$24.apply(Analyzer.scala:509)

Issue Links

• links to: [Github] Pull Request #17224 (HyukjinKwon)
• links to: [Github] Pull Request #17226 (aray)