[SPARK-47824] Nondeterminism in pyspark.pandas.series.asof

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.2, 4.0.0, 3.5.1, 3.3.4
• Fix Version/s: 4.0.0, 3.5.2, 3.4.3
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Mark Jarvin
• Reporter: Mark Jarvin
• Created: 11/Apr/24 23:18
• Updated: 12/Apr/24 00:38
• Resolved: 12/Apr/24 00:38

Description

`max_by` in `pyspark.pandas.series.asof` uses a literal string instead of a generated column as its ordering condition, resulting in nondeterminism.

Issue Links

• links to: GitHub Pull Request #46018