Hadoop Common
  1. Hadoop Common
  2. HADOOP-985

Namenode should identify DataNodes as ip:port instead of hostname:port

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      Right now NameNode keeps track of DataNodes with "hostname:port". One proposal is to keep track of datanodes with "ip:port". There are various concerns expressed regd hostnames and ip. Please add your experiences here so that we have better idea on what we should fix etc.

      How should be calculate datanode ip:

      1) Just like how we calculate hostname currently with "dfs.datanode.dns.interface" and "dfs.datanode.dns.nameserver". So if interface specified wrong, it could report ip like 127.0.0.1 which might or might not be intended.

      2) Namenode can use the remove socket address when the datanode registers. Not sure how easy it to get this address in RPC or if this is desirable.

      3) Namenode could just resolve the hostname when a datanode registers. It could print of a warning if the resolved ip and reported ip don't match.

      One advantage of using IPs is that DFSClient does not need to resolve them when it connects to datanode. This could save few milliseconds for each block. Also, DFSClient should check all its ips to see if a given ip is local or not.

      As far I see namenode does not resolve any DNS in normal operations since it does not actively contact datanodes. In that sense not sure if this have any change in Namenode performance.

      Thoughts?

      1. dfshealth.html
        4 kB
        Raghu Angadi
      2. HADOOP-985-1.patch
        14 kB
        Raghu Angadi
      3. HADOOP-985-2.patch
        14 kB
        Raghu Angadi
      4. HADOOP-985-3.patch
        23 kB
        Raghu Angadi
      5. HADOOP-985-4.patch
        23 kB
        Raghu Angadi
      6. HADOOP-985-5.patch
        25 kB
        Raghu Angadi
      7. HADOOP-985-6.patch
        25 kB
        Raghu Angadi

        Issue Links

          Activity

          Raghu Angadi created issue -
          Raghu Angadi made changes -
          Field Original Value New Value
          Assignee Raghu Angadi [ rangadi ]
          Raghu Angadi made changes -
          Affects Version/s 0.11.0 [ 12312257 ]
          Fix Version/s 0.12.0 [ 12312293 ]
          Component/s dfs [ 12310710 ]
          Raghu Angadi made changes -
          Link This issue relates to HADOOP-685 [ HADOOP-685 ]
          Raghu Angadi made changes -
          Attachment dfshealth.html [ 12350791 ]
          Raghu Angadi made changes -
          Link This issue incorporates HADOOP-697 [ HADOOP-697 ]
          Raghu Angadi made changes -
          Attachment HADOOP-985-1.patch [ 12350813 ]
          Raghu Angadi made changes -
          Attachment HADOOP-985-2.patch [ 12350814 ]
          Raghu Angadi made changes -
          Attachment HADOOP-985-3.patch [ 12351300 ]
          Raghu Angadi made changes -
          Attachment HADOOP-985-4.patch [ 12351304 ]
          Raghu Angadi made changes -
          Attachment HADOOP-985-5.patch [ 12351409 ]
          Raghu Angadi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Raghu Angadi made changes -
          Attachment HADOOP-985-6.patch [ 12351748 ]
          Raghu Angadi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Doug Cutting made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Doug Cutting made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HADOOP-6867 [ HADOOP-6867 ]

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development