[SPARK-2494] Hash of None is different cross machines in CPython

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0, 0.9.1, 0.9.2, 1.0.0, 1.0.1
• Fix Version/s: 0.9.3, 1.0.2, 1.1.0
• Component/s: PySpark
• Labels: pyspark shuffle

Environment

CPython 2.x

• Target Version/s: 0.9.3, 1.0.2, 1.1.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 15/Jul/14 19:18
• Updated: 21/Jul/14 19:04
• Resolved: 21/Jul/14 19:04

Description

The hash of None, also tuple with None in it, is different cross machines, so the result will be wrong if None appear in the key of partitionBy().

It should use an portable hash function as the default partition function, which generate same hash for all the builtin immutable types, especially tuple.

Issue Links

• links to: [Github] Pull Request #1371 (davies)