[SPARK-24374] SPIP: Support Barrier Execution Mode in Apache Spark

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: ML, Spark Core
• Labels: Hydrogen SPIP
• Epic Name: Support Barrier Execution Mode
• Target Version/s: 2.4.0
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Shepherd: Reynold Xin
• Created: 24/May/18 04:48
• Updated: 19/Mar/21 17:37
• Resolved: 19/Mar/21 17:37

Description

(See details in the linked/attached SPIP doc.)

The proposal here is to add a new scheduling model to Apache Spark so users can properly embed distributed DL training as a Spark stage to simplify the distributed training workflow. For example, Horovod uses MPI to implement all-reduce to accelerate distributed TensorFlow training. The computation model is different from MapReduce used by Spark. In Spark, a task in a stage doesn’t depend on any other tasks in the same stage, and hence it can be scheduled independently. In MPI, all workers start at the same time and pass messages around. To embed this workload in Spark, we need to introduce a new scheduling model, tentatively named “barrier scheduling”, which launches tasks at the same time and provides users enough information and tooling to embed distributed DL training. Spark can also provide an extra layer of fault tolerance in case some tasks failed in the middle, where Spark would abort all tasks and restart the stage.

Issue Links

• relates to: MAPREDUCE-2911 Hamster: Hadoop And Mpi on the same cluSTER
• relates to: SPARK-1485 Implement AllReduce
• relates to: SPARK-20928 SPIP: Continuous Processing Mode for Structured Streaming
• links to: SPIP (Google Doc)