Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3150

Add option for clients to contact DNs via hostname

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: 1.1.0, 2.0.2-alpha
    • Component/s: datanode, hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The DN listens on multiple IP addresses (the default dfs.datanode.address is the wildcard) however per HADOOP-6867 only the source address (IP) of the registration is given to clients. HADOOP-985 made clients access datanodes by IP primarily to avoid the latency of a DNS lookup, this had the side effect of breaking DN multihoming (the client can not route the IP exposed by the NN if the DN registers with an interface that has a cluster-private IP). To fix this let's add back the option for Datanodes to be accessed by hostname.

      This can be done by:

      1. Modifying the primary field of the Datanode descriptor to be the hostname, or
      2. Modifying Client/Datanode <-> Datanode access use the hostname field instead of the IP

      Approach #2 does not require an incompatible client protocol change, and is much less invasive. It minimizes the scope of modification to just places where clients and Datanodes connect, vs changing all uses of Datanode identifiers.

      New client and Datanode configuration options are introduced:

      • dfs.client.use.datanode.hostname indicates all client to datanode connections should use the datanode hostname (as clients outside cluster may not be able to route the IP)
      • dfs.datanode.use.datanode.hostname indicates whether Datanodes should use hostnames when connecting to other Datanodes for data transfer

      If the configuration options are not used, there is no change in the current behavior.

        Attachments

        1. hdfs-3150-b1.txt
          32 kB
          Eli Collins
        2. hdfs-3150-b1.txt
          32 kB
          Eli Collins
        3. hdfs-3150.txt
          50 kB
          Eli Collins
        4. hdfs-3150.txt
          50 kB
          Eli Collins
        5. hdfs-3150.txt
          50 kB
          Eli Collins
        6. hdfs-3150.txt
          50 kB
          Eli Collins

          Issue Links

            Activity

              People

              • Assignee:
                eli Eli Collins
                Reporter:
                eli2 Eli Collins
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: