Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5113

Audit and document use of hostnames and IP addresses in Spark

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Spark has multiple network components that start servers and advertise their network addresses to other processes.

      We should go through each of these components and make sure they have consistent and/or documented behavior wrt (a) what interface(s) they bind to and (b) what hostname they use to advertise themselves to other processes. We should document this clearly and explain to people what to do in different cases (e.g. EC2, dockerized containers, etc).

      When Spark initializes, it will search for a network interface until it finds one that is not a loopback address. Then it will do a reverse DNS lookup for a hostname associated with that interface. Then the network components will use that hostname to advertise the component to other processes. That hostname is also the one used for the akka system identifier (akka supports only supplying a single name which it uses both as the bind interface and as the actor identifier). In some cases, that hostname is used as the bind hostname also (e.g. I think this happens in the connection manager and possibly akka) - which will likely internally result in a re-resolution of this to an IP address. In other cases (the web UI and netty shuffle) we seem to bind to all interfaces.

      The best outcome would be to have three configs that can be set on each machine:

      SPARK_LOCAL_IP # Ip address we bind to for all services
      SPARK_INTERNAL_HOSTNAME # Hostname we advertise to remote processes within the cluster
      SPARK_EXTERNAL_HOSTNAME # Hostname we advertise to processes outside the cluster (e.g. the UI)
      

      It's not clear how easily we can support that scheme while providing backwards compatibility. The last one (SPARK_EXTERNAL_HOSTNAME) is easy - it's just an alias for what is now SPARK_PUBLIC_DNS.

        Issue Links

          Activity

          Hide
          farrellee Matthew Farrellee added a comment -

          Patrick Wendell would SPARK_INTERNAL_HOSTNAME operate like SPARK_LOCAL_HOSTNAME?

          Show
          farrellee Matthew Farrellee added a comment - Patrick Wendell would SPARK_INTERNAL_HOSTNAME operate like SPARK_LOCAL_HOSTNAME?
          Hide
          srowen Sean Owen added a comment -

          Not sure if this one will ever happen, but there's a related problem at SPARK-10149

          Show
          srowen Sean Owen added a comment - Not sure if this one will ever happen, but there's a related problem at SPARK-10149

            People

            • Assignee:
              Unassigned
              Reporter:
              pwendell Patrick Wendell
            • Votes:
              7 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development