[SPARK-48184] Always set the seed of dataframe.sample in Client side

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.1, 3.4.3
• Fix Version/s: 4.0.0, 3.5.2
• Component/s: Connect, PySpark
• Labels: pull-request-available
• Assignee: Ruifeng Zheng
• Reporter: Ruifeng Zheng
• Created: 08/May/24 03:16
• Updated: 08/May/24 23:44
• Resolved: 08/May/24 14:44

Description

the output dataframe of `sample` is not immutable in Spark Connect

 

In Spark Classic:

In [1]: df = spark.range(10000).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006]

 

In Spark Connect:

In [1]: df = spark.range(10000).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979]
 

 

 

Issue Links

• links to: GitHub Pull Request #46456