Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13103

Group resolution from LDAP may fail on javax.naming.ServiceUnavailableException

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      According to the javadoc, ServiceUnavailableException is thrown when attempting to communicate with a directory or naming service and that service is not available. It might be unavailable for different reasons. For example, the server might be too busy to service the request, or the server might not be registered to service any requests, etc.

      We should retry on it.

      1. c13103_20160505.patch
        4 kB
        Tsz Wo Nicholas Sze
      2. c13103_20160505b.patch
        5 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        c13103_20160505.patch: 1st patch.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - c13103_20160505.patch: 1st patch.
        Hide
        cnauroth Chris Nauroth added a comment -

        Hello Tsz Wo Nicholas Sze.

        Something that has been bugging me for a while is that we also don't support timeouts on these LDAP connections. This page has documentation on how to set the timeouts through the JNDI APIs.

        https://docs.oracle.com/javase/tutorial/jndi/newstuff/readtimeout.html

        Would you be interested in doing timeouts as part of this patch, or do you prefer if I file a separate JIRA for timeouts? I'm fine either way, and I can help code review whatever you decide to do here.

        Thanks!

        Show
        cnauroth Chris Nauroth added a comment - Hello Tsz Wo Nicholas Sze . Something that has been bugging me for a while is that we also don't support timeouts on these LDAP connections. This page has documentation on how to set the timeouts through the JNDI APIs. https://docs.oracle.com/javase/tutorial/jndi/newstuff/readtimeout.html Would you be interested in doing timeouts as part of this patch, or do you prefer if I file a separate JIRA for timeouts? I'm fine either way, and I can help code review whatever you decide to do here. Thanks!
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Hi Chris Nauroth, please file a separated JIRA for the timeouts. Thanks.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Hi Chris Nauroth , please file a separated JIRA for the timeouts. Thanks.
        Hide
        cnauroth Chris Nauroth added a comment -

        Sounds good. I filed HADOOP-13105.

        Show
        cnauroth Chris Nauroth added a comment - Sounds good. I filed HADOOP-13105 .
        Hide
        cnauroth Chris Nauroth added a comment -

        The patch looks good to me overall. I just have one question. The old code had one attempt outside the while loop and then up to 3 more attempts inside the while loop. With this patch, all attempts are collapsed into a single for loop with up to 3 iterations. I am wondering if RECONNECT_RETRY_COUNT should be increased to 4 to preserve the old behavior of "4 total attempts". Let me know your thoughts.

        Show
        cnauroth Chris Nauroth added a comment - The patch looks good to me overall. I just have one question. The old code had one attempt outside the while loop and then up to 3 more attempts inside the while loop. With this patch, all attempts are collapsed into a single for loop with up to 3 iterations. I am wondering if RECONNECT_RETRY_COUNT should be increased to 4 to preserve the old behavior of "4 total attempts". Let me know your thoughts.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 5m 43s trunk passed with JDK v1.8.0_91
        +1 compile 6m 35s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 35s trunk passed
        +1 javadoc 0m 53s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 41s the patch passed
        +1 compile 5m 41s the patch passed with JDK v1.8.0_91
        +1 javac 5m 41s the patch passed
        +1 compile 6m 39s the patch passed with JDK v1.7.0_95
        +1 javac 6m 39s the patch passed
        +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: The patch generated 0 new + 128 unchanged - 6 fixed = 128 total (was 134)
        +1 mvnsite 0m 57s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 55s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95
        -1 unit 6m 26s hadoop-common in the patch failed with JDK v1.8.0_91.
        -1 unit 6m 45s hadoop-common in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        57m 23s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.security.TestLdapGroupsMapping
        JDK v1.7.0_95 Failed junit tests hadoop.security.TestLdapGroupsMapping



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802489/c13103_20160505.patch
        JIRA Issue HADOOP-13103
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bdb96da3e82a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 35cf503
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 46s trunk passed +1 compile 5m 43s trunk passed with JDK v1.8.0_91 +1 compile 6m 35s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 0m 53s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 41s the patch passed +1 compile 5m 41s the patch passed with JDK v1.8.0_91 +1 javac 5m 41s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_95 +1 javac 6m 39s the patch passed +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: The patch generated 0 new + 128 unchanged - 6 fixed = 128 total (was 134) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 55s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95 -1 unit 6m 26s hadoop-common in the patch failed with JDK v1.8.0_91. -1 unit 6m 45s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 57m 23s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.security.TestLdapGroupsMapping JDK v1.7.0_95 Failed junit tests hadoop.security.TestLdapGroupsMapping Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802489/c13103_20160505.patch JIRA Issue HADOOP-13103 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bdb96da3e82a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 35cf503 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9294/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        > ... The old code had one attempt outside the while loop and then up to 3 more attempts inside the while loop. With this patch, all attempts are collapsed into a single for loop with up to 3 iterations. ...

        Good observation. I intentionally make such change so that it can fail fast. For the exception cases I saw, the first try may fail with various reasons (such as connection timeout). Then, the second try always success since it reestablishes the connection. The third try is almost redundant but let's give it a try. It is unlikely that the third try will success after the first and the second tries have failed.) The fourth try is just wasting time.

        Will update the patch to fix the unit test.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - > ... The old code had one attempt outside the while loop and then up to 3 more attempts inside the while loop. With this patch, all attempts are collapsed into a single for loop with up to 3 iterations. ... Good observation. I intentionally make such change so that it can fail fast. For the exception cases I saw, the first try may fail with various reasons (such as connection timeout). Then, the second try always success since it reestablishes the connection. The third try is almost redundant but let's give it a try. It is unlikely that the third try will success after the first and the second tries have failed.) The fourth try is just wasting time. Will update the patch to fix the unit test.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        c13103_20160505b.patch: fixes test failure.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - c13103_20160505b.patch: fixes test failure.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch, pending a fresh pre-commit run. Tsz Wo Nicholas Sze, thank you for the patch.

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch, pending a fresh pre-commit run. Tsz Wo Nicholas Sze , thank you for the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 38s trunk passed
        +1 compile 5m 57s trunk passed with JDK v1.8.0_91
        +1 compile 6m 47s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 37s trunk passed
        +1 javadoc 0m 54s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 5s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 42s the patch passed
        +1 compile 5m 52s the patch passed with JDK v1.8.0_91
        +1 javac 5m 52s the patch passed
        +1 compile 6m 49s the patch passed with JDK v1.7.0_95
        +1 javac 6m 49s the patch passed
        +1 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 0 new + 128 unchanged - 6 fixed = 128 total (was 134)
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 0m 55s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95
        +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_91.
        +1 unit 8m 13s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        61m 35s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802519/c13103_20160505b.patch
        JIRA Issue HADOOP-13103
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cfe50d546dfd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8faf47
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9296/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9296/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 38s trunk passed +1 compile 5m 57s trunk passed with JDK v1.8.0_91 +1 compile 6m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 54s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 5s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 42s the patch passed +1 compile 5m 52s the patch passed with JDK v1.8.0_91 +1 javac 5m 52s the patch passed +1 compile 6m 49s the patch passed with JDK v1.7.0_95 +1 javac 6m 49s the patch passed +1 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 0 new + 128 unchanged - 6 fixed = 128 total (was 134) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 55s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95 +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.8.0_91. +1 unit 8m 13s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 61m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802519/c13103_20160505b.patch JIRA Issue HADOOP-13103 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cfe50d546dfd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8faf47 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9296/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9296/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Thanks Chris for reviewing the patches.

        I have committed this.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Chris for reviewing the patches. I have committed this.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9726 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9726/)
        HADOOP-13103 Group resolution from LDAP may fail on (szetszwo: rev f305d9c0f64fd7d085f01eaae2154ef13b05b197)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMapping.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9726 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9726/ ) HADOOP-13103 Group resolution from LDAP may fail on (szetszwo: rev f305d9c0f64fd7d085f01eaae2154ef13b05b197) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMapping.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            szetszwo Tsz Wo Nicholas Sze
            Reporter:
            szetszwo Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development