Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently BackupNode uses DNS to find its default host name, and then starts RPC server listening on that address ignoring the address specified in the configuration. Therefore, there is no way to start BackupNode on a particular ip or host address. BackupNode should use the address specified in the configuration instead.

      1. BNAddress.patch
        2 kB
        Konstantin Shvachko
      2. BNAddress.patch
        11 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          BackupNode uses rpc, http and service rpc addresses as configured.

          Show
          Konstantin Shvachko added a comment - BackupNode uses rpc, http and service rpc addresses as configured.
          Hide
          Todd Lipcon added a comment -

          +1 pending test results

          Show
          Todd Lipcon added a comment - +1 pending test results
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12467147/BNAddress.patch
          against trunk revision 1055142.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
          org.apache.hadoop.hdfs.TestHDFSServerPorts

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/74//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/74//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/74//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12467147/BNAddress.patch against trunk revision 1055142. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.TestHDFSServerPorts -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/74//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/74//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/74//console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          TestStorageRestore is HDFS-1496.
          The rest is related to IPv6 problems. TestBackupNode and TestHDFSServerPorts reuse configuration after it has been updated by NN or BN. But the updated config contains IPv6-style address "0:0:0:0:0:0:0:0:50100" instead of "0.0.0.0:50100", which results in the following exception in NetUtils.

          java.lang.NumberFormatException: For input string: "0:0:0:0:0:0:0:50100"
          	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
          	at java.lang.Integer.parseInt(Integer.java:456)
          	at java.lang.Integer.parseInt(Integer.java:497)
          	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:155)
          	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:129)
          	at org.apache.hadoop.hdfs.server.namenode.BackupNode.getRpcServerAddress(BackupNode.java:81)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:350)
          	at org.apache.hadoop.hdfs.server.namenode.BackupNode.initialize(BackupNode.java:129)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:578)
          	at org.apache.hadoop.hdfs.server.namenode.BackupNode.<init>(BackupNode.java:72)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1532)
          	at org.apache.hadoop.hdfs.TestHDFSServerPorts.startBackupNode(TestHDFSServerPorts.java:137)
          	at org.apache.hadoop.hdfs.TestHDFSServerPorts.canStartBackupNode(TestHDFSServerPorts.java:236)
          	at org.apache.hadoop.hdfs.TestHDFSServerPorts.testBackupNodePorts(TestHDFSServerPorts.java:382)
          

          I can fix it by resetting config parameters, but may it is better to run Hudson with IPv4 flag as suggested in HADOOP-6056.
          What people think?

          Show
          Konstantin Shvachko added a comment - TestStorageRestore is HDFS-1496 . The rest is related to IPv6 problems. TestBackupNode and TestHDFSServerPorts reuse configuration after it has been updated by NN or BN. But the updated config contains IPv6-style address "0:0:0:0:0:0:0:0:50100" instead of "0.0.0.0:50100", which results in the following exception in NetUtils. java.lang.NumberFormatException: For input string: "0:0:0:0:0:0:0:50100" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang. Integer .parseInt( Integer .java:456) at java.lang. Integer .parseInt( Integer .java:497) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:155) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:129) at org.apache.hadoop.hdfs.server.namenode.BackupNode.getRpcServerAddress(BackupNode.java:81) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:350) at org.apache.hadoop.hdfs.server.namenode.BackupNode.initialize(BackupNode.java:129) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:578) at org.apache.hadoop.hdfs.server.namenode.BackupNode.<init>(BackupNode.java:72) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1532) at org.apache.hadoop.hdfs.TestHDFSServerPorts.startBackupNode(TestHDFSServerPorts.java:137) at org.apache.hadoop.hdfs.TestHDFSServerPorts.canStartBackupNode(TestHDFSServerPorts.java:236) at org.apache.hadoop.hdfs.TestHDFSServerPorts.testBackupNodePorts(TestHDFSServerPorts.java:382) I can fix it by resetting config parameters, but may it is better to run Hudson with IPv4 flag as suggested in HADOOP-6056 . What people think?
          Hide
          Konstantin Shvachko added a comment -

          I replaced default addresses with real ones in order to avoid IPv6 conversions.

          Show
          Konstantin Shvachko added a comment - I replaced default addresses with real ones in order to avoid IPv6 conversions.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12467861/BNAddress.patch
          against trunk revision 1056206.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.fs.permission.TestStickyBit
          org.apache.hadoop.hdfs.security.TestDelegationToken
          org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
          org.apache.hadoop.hdfs.TestCrcCorruption
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          org.apache.hadoop.hdfs.TestDatanodeDeath
          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.hdfs.TestDFSFinalize
          org.apache.hadoop.hdfs.TestDFSRollback
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestDFSStartupVersions
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
          org.apache.hadoop.hdfs.TestDFSUpgrade
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestFileAppend3
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestFileAppend
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart
          org.apache.hadoop.hdfs.TestFileCreation
          org.apache.hadoop.hdfs.TestHDFSFileSystemContract
          org.apache.hadoop.hdfs.TestHDFSTrash
          org.apache.hadoop.hdfs.TestPread
          org.apache.hadoop.hdfs.TestQuota
          org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.TestRestartDFS
          org.apache.hadoop.hdfs.TestSetrepDecreasing
          org.apache.hadoop.hdfs.TestSetrepIncreasing
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/95//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/95//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/95//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12467861/BNAddress.patch against trunk revision 1056206. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.fs.permission.TestStickyBit org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.TestDFSRollback org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDFSStartupVersions org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.hdfs.TestDFSUpgrade org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.TestPread org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.TestWriteConfigurationToDFS -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/95//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/95//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/95//console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          Everything timed out. Resubmitting and making it a blocker for 0.22

          Show
          Konstantin Shvachko added a comment - Everything timed out. Resubmitting and making it a blocker for 0.22
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12467861/BNAddress.patch
          against trunk revision 1059508.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/118//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/118//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/118//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12467861/BNAddress.patch against trunk revision 1059508. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery org.apache.hadoop.hdfs.server.namenode.TestStorageRestore -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/118//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/118//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/118//console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          TestHDFSCLI and TestBlockRecovery timed out on Hudson. I ran test target locally. Only TestStorageRestore fails.
          Going to commit this.

          Show
          Konstantin Shvachko added a comment - TestHDFSCLI and TestBlockRecovery timed out on Hudson. I ran test target locally. Only TestStorageRestore fails. Going to commit this.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.
          Hide
          Tom White added a comment -

          Looks like MR test compilation is failing now. I filed MAPREDUCE-2282.

          Show
          Tom White added a comment - Looks like MR test compilation is failing now. I filed MAPREDUCE-2282 .
          Hide
          Tom White added a comment -

          I looked at trying to fix this by replacing references to NAME_NODE_HTTP and NAME_NODE_HTTP_HOST with THIS_HOST (all in TestHDFSServerPorts), but THIS_HOST is not accessible from TestMRServerPorts. So should we revert this while we come up with a fix? Thanks.

          Show
          Tom White added a comment - I looked at trying to fix this by replacing references to NAME_NODE_HTTP and NAME_NODE_HTTP_HOST with THIS_HOST (all in TestHDFSServerPorts), but THIS_HOST is not accessible from TestMRServerPorts. So should we revert this while we come up with a fix? Thanks.
          Hide
          Konstantin Shvachko added a comment -

          Yes, go ahead revert/reopen, I'll check on mapreduce.

          Show
          Konstantin Shvachko added a comment - Yes, go ahead revert/reopen, I'll check on mapreduce.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Yes, go ahead revert/reopen, I'll check on mapreduce.

          Unfortunately, it seems the HDFS test codes, which are supposed to be unstable, are used in other projects. Not a big deal. Just fix it in either HDFS or MapReduce but not necessarily reverting this.

          Show
          Tsz Wo Nicholas Sze added a comment - > Yes, go ahead revert/reopen, I'll check on mapreduce. Unfortunately, it seems the HDFS test codes, which are supposed to be unstable, are used in other projects. Not a big deal. Just fix it in either HDFS or MapReduce but not necessarily reverting this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/ )

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development