Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22906

External shuffle IP different from Host ip

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 2.2.1
    • Fix Version/s: None
    • Component/s: Block Manager, Spark Core
    • Labels:

      Description

      Now is possible to configure the spark.shuffle.service.port, but there aren't an equivalent for host. Imagine that you are using an external shuffle deployed in Docker.

      If you aren't using Host mode for the docker, you may want to use the internal ip address of the docker to connect to the external shuffle service.

      Also you could use Calico, or just being attached the spark shuffle to a different IP that is used in the Spark executor (example hosts with multiple network interfaces).

      So this is why I implemented the spark.shuffle.service.host configuration

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pianista Unai Sarasola
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: