[SPARK-7982] crosstab should use 0 instead of null for pairs that don't appear

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SQL
• Labels: starter
• Target Version/s: 1.4.1
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 31/May/15 08:27
• Updated: 02/Jun/15 04:11
• Resolved: 02/Jun/15 04:11

Description

See the following

In [79]: sqlContext.range(0, 10).stat.crosstab('id', 'id').show()
+-----+----+----+----+----+----+----+----+----+----+----+
|id_id|   0|   5|   1|   6|   9|   2|   7|   3|   8|   4|
+-----+----+----+----+----+----+----+----+----+----+----+
|    0|   1|null|null|null|null|null|null|null|null|null|
|    5|null|null|null|null|null|   1|null|null|null|null|
|    1|null|   1|null|null|null|null|null|null|null|null|
|    6|null|null|null|null|null|null|   1|null|null|null|
|    9|null|null|null|null|null|null|null|null|null|   1|
|    2|null|null|   1|null|null|null|null|null|null|null|
|    7|null|null|null|null|null|null|null|   1|null|null|
|    3|null|null|null|   1|null|null|null|null|null|null|
|    8|null|null|null|null|null|null|null|null|   1|null|
|    4|null|null|null|null|   1|null|null|null|null|null|
+-----+----+----+----+----+----+----+----+----+----+----+

I think we should use 0 instead of null for these columns.

Issue Links

• links to: [Github] Pull Request #6566 (rxin)