[SPARK-24725] Discuss necessary info and access in barrier mode + Mesos

• Type: Story
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: ML, Spark Core
• Target Version/s: 3.0.0
• Epic Link: Support Barrier Execution Mode
• Assignee: Unassigned
• Reporter: Xiangrui Meng
• Created: 02/Jul/18 23:25
• Updated: 07/Jan/19 13:55

Description

In barrier mode, to run hybrid distributed DL training jobs, we need to provide users sufficient info and access so they can set up a hybrid distributed training job, e.g., using MPI.

This ticket limits the scope of discussion to Spark + Mesos. I'm not aware of MPI support in Mesos. So we should find someone with good knowledge to lead the discussion here.