[SPARK-8903] Fix NPE when cross tab elements contain nulls (branch 1.4)

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.4.1
• Fix Version/s: 1.4.1
• Component/s: SQL
• Target Version/s: 1.4.1
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 08/Jul/15 18:52
• Updated: 08/Jul/15 22:33
• Resolved: 08/Jul/15 22:33

Description

The backport of SPARK-8803 to branch-1.4 contains an error:

It calls

countsRow.setString(0, cleanElement(col1Item.toString))

instead of

countsRow.setString(0, cleanElement(col1Item))

which leads to a NullPointerException when col1Item == null.'

This caused a test failure in DataFrameStatSuite:

- special crosstab elements (., '', null, ``) *** FAILED ***
  java.lang.NullPointerException:
  at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131)
  at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
  at scala.collection.AbstractTraversable.map(Traversable.scala:105)
  at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121)
  at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94)
  at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97)
 

Issue Links

• links to: [Github] Pull Request #7295 (JoshRosen)