[SPARK-24581] Design: BarrierTaskContext.barrier()

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Component/s: ML, Spark Core
• Epic Link: Support Barrier Execution Mode
• Assignee: Xingbo Jiang
• Reporter: Xiangrui Meng
• Created: 18/Jun/18 15:41
• Updated: 20/Sep/18 17:23
• Resolved: 30/Jul/18 14:51

Description

We need to provide a communication barrier function to users to help coordinate tasks within a barrier stage. This is very similar to MPI_Barrier function in MPI. This story is for its design.

 

Requirements:

• Low-latency. The tasks should be unblocked soon after all tasks have reached this barrier. The latency is more important than CPU cycles here.
• Support unlimited timeout with proper logging. For DL tasks, it might take very long to converge, we should support unlimited timeout with proper logging. So users know why a task is waiting.