Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13422

ZKDelegationTokenSecretManager JaasConfig does not work well with other ZK users in process

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There's a race in the globals. The non-global APIs from ZOOKEEPER-2139 are not available yet in a stable ZK version and there's no timeline for availability, so for now it would help to make SM aware of other users of the global config.

      1. HADOOP-13422.01.patch
        3 kB
        Sergey Shelukhin
      2. HADOOP-13422.patch
        1 kB
        Sergey Shelukhin

        Activity

        Hide
        sershe Sergey Shelukhin added a comment -

        The initial patch.

        Show
        sershe Sergey Shelukhin added a comment - The initial patch.
        Hide
        cnauroth Chris Nauroth added a comment -
        	private final javax.security.auth.login.Configuration baseConfig = javax.security.auth.login.Configuration
        	        .getConfiguration();
        

        I expect pre-commit will flag a nitpick about indentation and line length exceeding 80 characters here, so we'll need one more patch revision.

        I'm in favor of the approach though. This will help avoid some bugs until we can implement a long-term fix that makes use of ZOOKEEPER-2139. There is already similar working code in Hive. (See the LlapZookeeperRegistryImpl class.) I know Sergey was able to demonstrate that this fix works through manual testing.

        Arun Suresh, are you interested in reviewing this? I'll give it some time before I consider committing.

        Show
        cnauroth Chris Nauroth added a comment - private final javax.security.auth.login.Configuration baseConfig = javax.security.auth.login.Configuration .getConfiguration(); I expect pre-commit will flag a nitpick about indentation and line length exceeding 80 characters here, so we'll need one more patch revision. I'm in favor of the approach though. This will help avoid some bugs until we can implement a long-term fix that makes use of ZOOKEEPER-2139 . There is already similar working code in Hive. (See the LlapZookeeperRegistryImpl class.) I know Sergey was able to demonstrate that this fix works through manual testing. Arun Suresh , are you interested in reviewing this? I'll give it some time before I consider committing.
        Hide
        asuresh Arun Suresh added a comment -

        Sure, will take a look later today...

        Show
        asuresh Arun Suresh added a comment - Sure, will take a look later today...
        Hide
        asuresh Arun Suresh added a comment -

        Thanks for the patch Sergey Shelukhin.. it looks straighforward and I understand unit testing this is non-trivial.

        Can you also modify ZKSignerSecretProvider to include this fix as well ? since ZKDTSM and the signerSecretProvider are generally used in conjunction.

        Show
        asuresh Arun Suresh added a comment - Thanks for the patch Sergey Shelukhin .. it looks straighforward and I understand unit testing this is non-trivial. Can you also modify ZKSignerSecretProvider to include this fix as well ? since ZKDTSM and the signerSecretProvider are generally used in conjunction.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 53s trunk passed
        +1 compile 7m 2s trunk passed
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 58s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 20s trunk passed
        +1 javadoc 0m 45s trunk passed
        +1 mvninstall 0m 38s the patch passed
        +1 compile 6m 42s the patch passed
        +1 javac 6m 42s the patch passed
        -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 4 new + 10 unchanged - 0 fixed = 14 total (was 10)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch 3 line(s) with tabs.
        +1 findbugs 1m 33s the patch passed
        +1 javadoc 0m 48s the patch passed
        +1 unit 8m 21s hadoop-common in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        39m 12s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820041/HADOOP-13422.patch
        JIRA Issue HADOOP-13422
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 208b135301e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 703fdf8
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/artifact/patchprocess/whitespace-tabs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 53s trunk passed +1 compile 7m 2s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 20s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 6m 42s the patch passed +1 javac 6m 42s the patch passed -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 4 new + 10 unchanged - 0 fixed = 14 total (was 10) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch 3 line(s) with tabs. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 8m 21s hadoop-common in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 39m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820041/HADOOP-13422.patch JIRA Issue HADOOP-13422 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 208b135301e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 703fdf8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/artifact/patchprocess/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10079/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sershe Sergey Shelukhin added a comment -

        Updated the patch.

        Show
        sershe Sergey Shelukhin added a comment - Updated the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 13s Maven dependency ordering for branch
        +1 mvninstall 14m 4s trunk passed
        +1 compile 14m 23s trunk passed
        +1 checkstyle 0m 46s trunk passed
        +1 mvnsite 2m 14s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 3m 5s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 10s the patch passed
        +1 compile 9m 59s the patch passed
        +1 javac 9m 59s the patch passed
        -0 checkstyle 0m 35s hadoop-common-project: The patch generated 2 new + 24 unchanged - 0 fixed = 26 total (was 24)
        +1 mvnsite 1m 43s the patch passed
        +1 mvneclipse 0m 41s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 51s the patch passed
        +1 javadoc 1m 15s the patch passed
        +1 unit 3m 37s hadoop-auth in the patch passed.
        -1 unit 11m 37s hadoop-common in the patch failed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        77m 18s



        Reason Tests
        Failed junit tests hadoop.security.TestGroupsCaching
          hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820274/HADOOP-13422.01.patch
        JIRA Issue HADOOP-13422
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0455f7d213d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d2cf8b5
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/testReport/
        modules C: hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common U: hadoop-common-project
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 14m 4s trunk passed +1 compile 14m 23s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 2m 14s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 10s the patch passed +1 compile 9m 59s the patch passed +1 javac 9m 59s the patch passed -0 checkstyle 0m 35s hadoop-common-project: The patch generated 2 new + 24 unchanged - 0 fixed = 26 total (was 24) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 51s the patch passed +1 javadoc 1m 15s the patch passed +1 unit 3m 37s hadoop-auth in the patch passed. -1 unit 11m 37s hadoop-common in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 77m 18s Reason Tests Failed junit tests hadoop.security.TestGroupsCaching   hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820274/HADOOP-13422.01.patch JIRA Issue HADOOP-13422 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0455f7d213d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d2cf8b5 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/testReport/ modules C: hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10090/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for the patch. The test failures from the last pre-commit run were unrelated. Checkstyle flagged some warnings about indentation level, which I can clean up trivially at commit time. I will commit this soon.

        Show
        cnauroth Chris Nauroth added a comment - +1 for the patch. The test failures from the last pre-commit run were unrelated. Checkstyle flagged some warnings about indentation level, which I can clean up trivially at commit time. I will commit this soon.
        Hide
        cnauroth Chris Nauroth added a comment -

        I have committed this to trunk, branch-2 and branch-2.8. Sergey Shelukhin, thank you for the patch. Arun Suresh, thank you for reviewing and pointing out the similar issue in ZKSignerSecretProvider.

        Show
        cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-2 and branch-2.8. Sergey Shelukhin , thank you for the patch. Arun Suresh , thank you for reviewing and pointing out the similar issue in ZKSignerSecretProvider .
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10157 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10157/)
        HADOOP-13422. ZKDelegationTokenSecretManager JaasConfig does not work (cnauroth: rev 255ea45e50e102505ee61eb0ba45ea93035abe3c)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
        • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10157 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10157/ ) HADOOP-13422 . ZKDelegationTokenSecretManager JaasConfig does not work (cnauroth: rev 255ea45e50e102505ee61eb0ba45ea93035abe3c) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java

          People

          • Assignee:
            sershe Sergey Shelukhin
            Reporter:
            sershe Sergey Shelukhin
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development