Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3171

The DatanodeID "name" field is overloaded

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The DatanodeID "name" field is currently overloaded, when the DN creates a DatanodeID to register with the NN it sets "name" to be the datanode hostname, which is the DN's "hostName" member. This isnot necesarily a FQDN, it is either set explicitly or determined by the DNS class, which could return the machine's hostname or the result of a DNS lookup, if configured to do so. The NN then clobbers the "name" field of the DatanodeID with the IP part of the new DatanodeID "name" field it creates (and sets the DatanodeID "hostName" field to the reported "name"). The DN gets the DatanodeID back from the NN and clobbers its "hostName" member with the "name" field of the returned DatanodeID. This makes the code hard to reason about eg DN#getMachine name sometimes returns a hostname and sometimes not, depending on when it's called in sequence with the registration. Ditto for uses of the "name" field. I think these contortions were originally performed because the DatanodeID didn't have a hostName field (it was part of DatanodeInfo) and so there was no way to communicate both at the same time. Now that the hostName field is in DatanodeID (as of HDFS-3164) we can establish the invariant that the "name" field always and only has an IP address and the "hostName" field always and only has a hostname.

      In HDFS-3144 I'm going to rename the "name" field so its clear that it contains an IP address. The above is enough scope for one change.

      1. hdfs-3171.txt
        44 kB
        Eli Collins

        Issue Links

          Activity

          Eli Collins created issue -
          Eli Collins made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-2609 [ HDFS-2609 ]
          Eli Collins made changes -
          Attachment hdfs-3171.txt [ 12520803 ]
          Eli Collins made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Eli Collins made changes -
          Attachment hdfs-3171.txt [ 12520803 ]
          Eli Collins made changes -
          Attachment hdfs-3171.txt [ 12520805 ]
          Eli Collins made changes -
          Link This issue blocks HDFS-3144 [ HDFS-3144 ]
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 2.0.0 [ 12320353 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Resolution Fixed [ 1 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HDFS-3164 [ HDFS-3164 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HADOOP-8348 [ HADOOP-8348 ]
          Eli Collins made changes -
          Assignee Eli Collins [ eli2 ] Eli Collins [ eli ]
          Gavin made changes -
          Link This issue blocks HDFS-3144 [ HDFS-3144 ]
          Gavin made changes -
          Link This issue is depended upon by HDFS-3144 [ HDFS-3144 ]

            People

            • Assignee:
              Eli Collins
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development