[SPARK-24726] Discuss necessary info and access in barrier mode + Standalone

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Affects Version/s: 3.0.0
• Component/s: ML, Spark Core
• Target Version/s: 2.4.0
• Epic Link: Support Barrier Execution Mode
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 02/Jul/18 23:43
• Updated: 20/Sep/18 17:16
• Resolved: 20/Jul/18 00:13

Description

In barrier mode, to run hybrid distributed DL training jobs, we need to provide users sufficient info and access so they can set up a hybrid distributed training job, e.g., using MPI.

This ticket limits the scope of discussion to Spark + Standalone. For MPI, what we need is password-less SSH access among workers. We might also consider other distributed frameworks, like distributed tensorflow, H2O, etc.