[SPARK-8621] crosstab exception when one of the value is empty

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: SQL
• Target Version/s: 1.4.1, 1.5.0
• Assignee: Wenchen Fan
• Reporter: Reynold Xin
• Created: 25/Jun/15 05:44
• Updated: 02/Jul/15 22:40
• Resolved: 01/Jul/15 17:31

Description

I think this happened because some value is empty.

scala> df1.stat.crosstab("role", "lang")
org.apache.spark.sql.AnalysisException: syntax error in attribute name: ;
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.parseAttributeName(LogicalPlan.scala:145)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveQuoted(LogicalPlan.scala:135)
	at org.apache.spark.sql.DataFrame.resolve(DataFrame.scala:157)
	at org.apache.spark.sql.DataFrame.col(DataFrame.scala:603)
	at org.apache.spark.sql.DataFrameNaFunctions.org$apache$spark$sql$DataFrameNaFunctions$$fillCol(DataFrameNaFunctions.scala:394)
	at org.apache.spark.sql.DataFrameNaFunctions$$anonfun$2.apply(DataFrameNaFunctions.scala:160)
	at org.apache.spark.sql.DataFrameNaFunctions$$anonfun$2.apply(DataFrameNaFunctions.scala:157)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
	at org.apache.spark.sql.DataFrameNaFunctions.fill(DataFrameNaFunctions.scala:157)
	at org.apache.spark.sql.DataFrameNaFunctions.fill(DataFrameNaFunctions.scala:147)
	at org.apache.spark.sql.DataFrameNaFunctions.fill(DataFrameNaFunctions.scala:132)
	at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:132)
	at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:91)

Issue Links

• relates to: SPARK-8803 Crosstab element's can't contain null's and back ticks
• links to: [Github] Pull Request #7117 (animeshbaranawal)
• links to: [Github] Pull Request #7149 (cloud-fan)