Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3688

Namenode loses datanode hostname if datanode re-registers

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      If a datanode ever re-registers with the namenode (e.g.: namenode restart, temporary network cut, etc.) then the datanode ends up registering with an IP address as the datanode name rather than the hostname.

        Issue Links

          Activity

          Hide
          Jason Lowe added a comment -

          When the datanode registers with the namenode, the namenode bashes the datanode's provided hostname with an IP address and sends this new registration object back to the datanode, which proceeds to bash over its name with the IP address. If the datanode ever re-registers, it uses the "updated" hostname (now IP address) when registering, and the hostname for the datanode is lost.

          This appears to only affect 0.23, since branch-2 and trunk have separate fields for tracking the datanode hostname vs. IP address, so the hostname isn't clobbered. See changes such as HDFS-3171, HDFS-3144, and HDFS-3164.

          Show
          Jason Lowe added a comment - When the datanode registers with the namenode, the namenode bashes the datanode's provided hostname with an IP address and sends this new registration object back to the datanode, which proceeds to bash over its name with the IP address. If the datanode ever re-registers, it uses the "updated" hostname (now IP address) when registering, and the hostname for the datanode is lost. This appears to only affect 0.23, since branch-2 and trunk have separate fields for tracking the datanode hostname vs. IP address, so the hostname isn't clobbered. See changes such as HDFS-3171 , HDFS-3144 , and HDFS-3164 .
          Hide
          Eli Collins added a comment -

          See also HDFS-3224, there's a bug in the check for DN re-registration with different storage ID.

          Show
          Eli Collins added a comment - See also HDFS-3224 , there's a bug in the check for DN re-registration with different storage ID.
          Hide
          Jason Lowe added a comment -

          Yes, ran into that as well when looking into this bug. Thanks for pointing out the JIRA, as it saves me from filing the duplicate.

          Show
          Jason Lowe added a comment - Yes, ran into that as well when looking into this bug. Thanks for pointing out the JIRA, as it saves me from filing the duplicate.
          Hide
          Jason Lowe added a comment -

          Patch for branch-0.23 that has the datanode preserve the original hostname when registering with the namenode.

          Show
          Jason Lowe added a comment - Patch for branch-0.23 that has the datanode preserve the original hostname when registering with the namenode.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537365/HDFS-3688.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2877//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537365/HDFS-3688.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2877//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Patch failed on trunk since it's only for 23. Manually ran test-patch for this on branch-0.23, here's the output:

          -1 overall.  
          
              +1 @author.  The patch does not contain any @author tags.
          
              +1 tests included.  The patch appears to include 3 new or modified tests.
          
              +1 javadoc.  The javadoc tool did not generate any warning messages.
          
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
              +1 eclipse:eclipse.  The patch built with eclipse:eclipse.
          
              +1 findbugs.  The patch does not introduce any new Findbugs (version ) warnings.
          
              +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
              -1 core tests.  The patch failed these unit tests:
                            org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          
              +1 contrib tests.  The patch passed contrib unit tests.
          

          The test failure appears to be unrelated, and when I ran the test again manually it passed. I believe this test is known to have some races in it, see HDFS-2881.

          Show
          Jason Lowe added a comment - Patch failed on trunk since it's only for 23. Manually ran test-patch for this on branch-0.23, here's the output: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. The test failure appears to be unrelated, and when I ran the test again manually it passed. I believe this test is known to have some races in it, see HDFS-2881 .
          Hide
          Daryn Sharp added a comment -

          +1 Looks good.

          Show
          Daryn Sharp added a comment - +1 Looks good.
          Hide
          Daryn Sharp added a comment -

          I've committed to branch-23. To reiterate Jason's earlier comment, issue has already been fixed in later versions. Thanks Jason!

          Show
          Daryn Sharp added a comment - I've committed to branch-23. To reiterate Jason's earlier comment, issue has already been fixed in later versions. Thanks Jason!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #323 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/323/)
          HDFS-3688. Namenode loses datanode hostname if datanode re-registers (Jason Lowe via daryn) (Revision 1364662)

          Result = UNSTABLE
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364662
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeRegister.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #323 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/323/ ) HDFS-3688 . Namenode loses datanode hostname if datanode re-registers (Jason Lowe via daryn) (Revision 1364662) Result = UNSTABLE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364662 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeRegister.java

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development