[SPARK-36705] Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Shuffle
• Target Version/s: 3.2.0
• Assignee: Minchu Yang
• Reporter: Mridul Muralidharan
• Created: 09/Sep/21 17:23
• Updated: 30/Sep/21 19:22
• Resolved: 13/Sep/21 21:16

Description

Push based shuffle is not compatible with io encryption or non-relocatable serialization.
This is similar to SPARK-34790

We have to disable push based shuffle if either of these two are true.

Issue Links

• links to: [Github] Pull Request #33976 (rmcyang)
• links to: [Github] Pull Request #33984 (rmcyang)
• links to: [Github] Pull Request #33987 (c21)
• links to: [Github] Pull Request #33987 (c21)
• links to: [Github] Pull Request #34158 (rmcyang)