Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28778

Shuffle jobs fail due to incorrect advertised address when running in virtual network

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.3, 2.3.0, 2.4.3
    • Fix Version/s: 2.4.5, 3.0.0
    • Component/s: Mesos
    • Labels:

      Description

      When shuffle jobs are launched by Mesos in a virtual network, Mesos scheduler sets executor --hostname parameter to 0.0.0.0 in the case when spark.mesos.network.name is provided. This makes executors use 0.0.0.0 as their advertised address and, in the presence of shuffle, executors fail to fetch shuffle blocks from each other using 0.0.0.0 as the origin. When a virtual network is used the hostname or IP address is not known upfront and assigned to a container at its start time so the executor process needs to advertise the correct dynamically assigned address to be reachable by other executors.

       

      The bug described above prevents Mesos users from running any jobs which involve shuffle due to the inability of executors to fetch shuffle blocks because of incorrect advertised address when virtual network is used.

        Attachments

          Activity

            People

            • Assignee:
              anton.kirillov Anton Kirillov
              Reporter:
              anton.kirillov Anton Kirillov
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: