[SPARK-38160] Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend

• Type: Bug
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: SQL
• Target Version/s: 3.3.0
• Assignee: Unassigned
• Reporter: EdisonWang
• Created: 09/Feb/22 08:48
• Updated: 09/Feb/22 10:26

Description

When we do shuffle on indeterminate expressions such as rand, and ShuffleFetchFailed happend, we may get incorrent result since it only retries failed map tasks.

We try to fix this by retry all upstream map tasks in this situation.

Issue Links

• links to: [Github] Pull Request #35460 (WangGuangxin)