[SPARK-9691] PySpark SQL rand function treats seed 0 as no seed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.4.1, 1.4.2, 1.5.0
• Fix Version/s: 1.4.2, 1.5.0
• Component/s: PySpark, SQL
• Target Version/s: 1.4.2, 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Yin Huai
• Reporter: Joseph K. Bradley
• Created: 06/Aug/15 17:30
• Updated: 07/Aug/15 00:28
• Resolved: 07/Aug/15 00:21

Description

In PySpark SQL's rand() function, it tests for a seed in a way such that seed 0 is treated as no seed, leading to non-deterministic results when a user would expect deterministic results.

See: https://github.com/apache/spark/blob/98e69467d4fda2c26a951409b5b7c6f1e9345ce4/python/pyspark/sql/functions.py#L271

Issue Links

• relates to: SPARK-9690 Add random seed Param to PySpark CrossValidator
• links to: [Github] Pull Request #7999 (yhuai)