[SPARK-20629] Copy shuffle data when nodes are being shut down

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: Kubernetes, Spark Core
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 07/May/17 03:38
• Updated: 01/Aug/23 21:40
• Resolved: 20/Jul/20 04:37

Description

We decided not to do this for YARN, but for Kubernetes and similar systems nodes may be shut down entirely without the ability to keep an AuxiliaryService around.

Issue Links

• Blocked: SPARK-31329 Modify executor monitor to allow for moving shuffle blocks
• causes: SPARK-44588 Migrated shuffle blocks are encrypted multiple times when io.encryption is enabled
• causes: SPARK-35589 BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating
• links to: [Github] Pull Request #28331 (holdenk)
• links to: [Github] Pull Request #28708 (holdenk)