[SPARK-29042] Sampling-based RDD with unordered input should be INDETERMINATE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Spark Core
• Labels: correctness
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 11/Sep/19 01:15
• Updated: 12/Dec/22 18:10
• Resolved: 13/Sep/19 21:07

Description

We have found and fixed the correctness issue when RDD output is INDETERMINATE. One missing part is sampling-based RDD. This kind of RDDs is order sensitive to its input. A sampling-based RDD with unordered input, should be INDETERMINATE.

Issue Links

• is related to: SPARK-23243 Shuffle+Repartition on an RDD could lead to incorrect answers
• links to: GitHub Pull Request #25751
• links to: GitHub Pull Request #25826