Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4448

Allow HA NN to start in secure mode with wildcard address configured

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.8.0
    • Component/s: ha, namenode, security
    • Labels:
      None

      Description

      Currently if one tries to configure HA NNs use the wildcard HTTP address when security is enabled, the NN will fail to start with an error like the following:

      java.lang.IllegalArgumentException: java.io.IOException: Cannot use a wildcard address with security. Must explicitly set bind address for Kerberos
      

      This is the case even if one configures an actual address for the other NN's HTTP address. There's no good reason for this, since we now check for the local address being set to 0.0.0.0 and determine the canonical hostname for Kerberos purposes using InetAddress.getLocalHost().getCanonicalHostName(), so we should remove the restriction.

      1. HDFS-4448.2.patch
        1 kB
        Arun Suresh
      2. HDFS-4448.patch
        2 kB
        Aaron T. Myers
      3. HDFS-4448.patch
        2 kB
        Aaron T. Myers

        Activity

        Hide
        Aaron T. Myers added a comment -

        Here's a patch which addresses the issue by simply removing the check which is now overly-restrictive.

        No tests are included since to test this adequately one needs multiple hosts and security to be enabled. I tested this patch on a secure 2-node HA cluster where each NN is configured itself to bind to 0.0.0.0, but is configured with an actual address for the other node. I confirmed that everything started up and checkpointing works as expected.

        Show
        Aaron T. Myers added a comment - Here's a patch which addresses the issue by simply removing the check which is now overly-restrictive. No tests are included since to test this adequately one needs multiple hosts and security to be enabled. I tested this patch on a secure 2-node HA cluster where each NN is configured itself to bind to 0.0.0.0, but is configured with an actual address for the other node. I confirmed that everything started up and checkpointing works as expected.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566904/HDFS-4448.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3901//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3901//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3901//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566904/HDFS-4448.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3901//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3901//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3901//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Whoops, left an unused variable in the previous patch. This patch is to address that findbugs warning.

        Show
        Aaron T. Myers added a comment - Whoops, left an unused variable in the previous patch. This patch is to address that findbugs warning.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566921/HDFS-4448.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3902//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3902//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566921/HDFS-4448.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3902//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3902//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        Does this present any issues for kerberos authentication on the multiple interfaces? I think we'd need principals for each canonical hostname of all interfaces, which I'm not sure the security infrastructure will support? Although perhaps I'm misunderstanding the issue.

        Show
        Daryn Sharp added a comment - Does this present any issues for kerberos authentication on the multiple interfaces? I think we'd need principals for each canonical hostname of all interfaces, which I'm not sure the security infrastructure will support? Although perhaps I'm misunderstanding the issue.
        Hide
        Aaron T. Myers added a comment -

        That's a great point, Daryn, and I agree with your analysis. Even though this patch will allow the NNs to start and function properly when bound to the wildcard address, clients (or DNs) will not in fact be able to connect on any interface not contained in the principal name used by the RPC server of the NN. A proper fix for this is thus somewhat more involved than I had originally anticipated.

        Show
        Aaron T. Myers added a comment - That's a great point, Daryn, and I agree with your analysis. Even though this patch will allow the NNs to start and function properly when bound to the wildcard address, clients (or DNs) will not in fact be able to connect on any interface not contained in the principal name used by the RPC server of the NN. A proper fix for this is thus somewhat more involved than I had originally anticipated.
        Hide
        Arun Suresh added a comment -

        Daryn Sharp, Aaron T. Myers It looks like after HADOOP-9789 , clients can be configured with dfs.namenode.kerberos.principal.pattern to make it accept a SPN that is different from the connecting namenode address.

        But NN still complains when starting up due to this check, which is now clearly not required. I plan to rebase and commit this within a day if there are no objections.

        Show
        Arun Suresh added a comment - Daryn Sharp , Aaron T. Myers It looks like after HADOOP-9789 , clients can be configured with dfs.namenode.kerberos.principal.pattern to make it accept a SPN that is different from the connecting namenode address. But NN still complains when starting up due to this check, which is now clearly not required. I plan to rebase and commit this within a day if there are no objections.
        Hide
        Arun Suresh added a comment -

        Rebasing against trunk..

        Show
        Arun Suresh added a comment - Rebasing against trunk..
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 22s There were no new javac warning messages.
        +1 javadoc 9m 40s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 5m 28s There were no new checkstyle issues.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 12s Pre-build of native portion
        +1 hdfs tests 163m 56s Tests passed in hadoop-hdfs.
            209m 48s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12727482/HDFS-4448.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 0ebe84d
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10354/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10354/testReport/
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10354//console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 22s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 5m 28s There were no new checkstyle issues. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 12s Pre-build of native portion +1 hdfs tests 163m 56s Tests passed in hadoop-hdfs.     209m 48s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727482/HDFS-4448.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0ebe84d hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10354/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10354/testReport/ Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10354//console This message was automatically generated.
        Hide
        Arun Suresh added a comment -

        No tests were included since as Aaron T. Myers had mentioned, Tested this manually on a 4 node secure-cluster and ensured both NNs came up and DNs were able to talk to them.

        Show
        Arun Suresh added a comment - No tests were included since as Aaron T. Myers had mentioned, Tested this manually on a 4 node secure-cluster and ensured both NNs came up and DNs were able to talk to them.
        Hide
        Arun Suresh added a comment -

        +1
        Committed to trunk and branch-2.
        Thanks Aaron T. Myers

        Show
        Arun Suresh added a comment - +1 Committed to trunk and branch-2. Thanks Aaron T. Myers
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7645/)
        HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7645/ ) HDFS-4448 . Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2104 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2104/)
        HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2104 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2104/ ) HDFS-4448 . Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/163/)
        HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/163/ ) HDFS-4448 . Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/172/)
        HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/172/ ) HDFS-4448 . Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #906 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/906/)
        HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #906 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/906/ ) HDFS-4448 . Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/173/)
        HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/173/ ) HDFS-4448 . Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2122/)
        HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2122/ ) HDFS-4448 . Allow HA NN to start in secure mode with wildcard address configured (atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java

          People

          • Assignee:
            Aaron T. Myers
            Reporter:
            Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development