Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30667

Support simple all gather in barrier task context

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: PySpark, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      Currently we offer task.barrier() to coordinate tasks in barrier mode. Tasks can see all IP addresses from BarrierTaskContext. It would be simpler to integrate with distributed frameworks like TensorFlow DistributionStrategy if we provide all gather that can let tasks share additional information with others, e.g., an available port.

      Note that with all gather, tasks are share their IP addresses as well.

      port = ... # get an available port
      ports = context.all_gather(port) # get all available ports, ordered by task ID
      ...  # set up distributed training service
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sarthfrey Sarth Frey
                Reporter:
                mengxr Xiangrui Meng
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: