[SPARK-17519] [MESOS] Enhance robustness when ExternalShuffleService is broken

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: Mesos
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Sun Rui
• Created: 13/Sep/16 07:36
• Updated: 21/May/19 04:16
• Resolved: 21/May/19 04:16

Description

This is intended to be a complement to SPARK-17370 which addressed Standalone mode only.
For Mesos, it seems we could enhance MesosExternalShuffleClient to detect if any of the external shuffle services is lost when sending heartbeats. In such case, the MesosCoarseGrainedSchedulerBackend can notify ExecutorLost with workerlost=true. Also it can put the slave where the external shuffle service run to the blacklist, preventing launching tasks further on it.

Issue Links

• is related to: SPARK-17370 Shuffle service files not invalidated when a slave is lost
• is related to: SPARK-20832 Standalone master should explicitly inform drivers of worker deaths and invalidate external shuffle service outputs