Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11973

Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: security
    • Labels:
      None

      Description

      I recently added an ACL Provider to the curator framework instance I pass to the ZkDelegationTokenSecretManager, and notice some strangeness around ACLs.

      I set: "zk-dt-secret-manager.znodeWorkingPath" to:
      "solr/zkdtsm"

      and notice that
      /solr/zkdtsm/
      /solr/zkdtsm/ZKDTSMRoot
      do not have ACLs

      but all the znodes under /solr/zkdtsm/ZKDTSMRoot have ACLs. From adding some logging, it looks like the ACLProvider is never called for /solr/zkdtsm and /solr/zkdtsm/ZKDTSMRoot. I don't know if that's a Curator or ZkDelegationTokenSecretManager issue.

      1. HADOOP-11973v3.patch
        6 kB
        Gregory Chanan
      2. HADOOP-11973v2.patch
        6 kB
        Gregory Chanan
      3. HADOOP-11973.patch
        6 kB
        Gregory Chanan

        Issue Links

          Activity

          Hide
          gchanan Gregory Chanan added a comment -

          any ideas Arun Suresh?

          Show
          gchanan Gregory Chanan added a comment - any ideas Arun Suresh ?
          Hide
          gchanan Gregory Chanan added a comment -

          looks like the underlying issue is CURATOR-221, I'm investigating if there is a workaround we can use.

          Show
          gchanan Gregory Chanan added a comment - looks like the underlying issue is CURATOR-221 , I'm investigating if there is a workaround we can use.
          Hide
          gchanan Gregory Chanan added a comment -

          Here's a patch that addresses the issue and has a test.

          Here's a description I wrote in CURATOR-221:

          Yes, although in my case it's a bit complicated. If you look at HADOOP-11973, to keep the external vs internal client impl similar, I want to initialize the final CuratorFramework object in the constructor, which means I want to use the namespace-aware version. So, I could create the nodes before I call usingNamespace, but then I have to deal with exception handling, which I don't want to do in the constructor. So essentially I have to do:

          call usingNamespace(ns) in the constructor
          in startThreads, call usingNamespace(null) and then create the parents manually.

          Show
          gchanan Gregory Chanan added a comment - Here's a patch that addresses the issue and has a test. Here's a description I wrote in CURATOR-221 : Yes, although in my case it's a bit complicated. If you look at HADOOP-11973 , to keep the external vs internal client impl similar, I want to initialize the final CuratorFramework object in the constructor, which means I want to use the namespace-aware version. So, I could create the nodes before I call usingNamespace, but then I have to deal with exception handling, which I don't want to do in the constructor. So essentially I have to do: call usingNamespace(ns) in the constructor in startThreads, call usingNamespace(null) and then create the parents manually.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 28s There were no new javac warning messages.
          +1 javadoc 9m 31s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 6s The applied patch generated 2 new checkstyle issues (total was 9, now 11).
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 1s Tests failed in hadoop-common.
              58m 56s  



          Reason Tests
          Failed unit tests hadoop.security.token.delegation.TestZKDelegationTokenSecretManager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733656/HADOOP-11973.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0790275
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 31s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 6s The applied patch generated 2 new checkstyle issues (total was 9, now 11). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 1s Tests failed in hadoop-common.     58m 56s   Reason Tests Failed unit tests hadoop.security.token.delegation.TestZKDelegationTokenSecretManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733656/HADOOP-11973.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0790275 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6734/console This message was automatically generated.
          Hide
          gchanan Gregory Chanan added a comment -

          Fix whitespace/style and unset thread local at end of test so other tests are not affected.

          Show
          gchanan Gregory Chanan added a comment - Fix whitespace/style and unset thread local at end of test so other tests are not affected.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the patch Gregory Chanan,

          +1 pending Jenkins

          Show
          asuresh Arun Suresh added a comment - Thanks for the patch Gregory Chanan , +1 pending Jenkins
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 44s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 6s The applied patch generated 1 new checkstyle issues (total was 9, now 10).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 24s Tests passed in hadoop-common.
              59m 55s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733678/HADOOP-11973v2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0790275
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 6s The applied patch generated 1 new checkstyle issues (total was 9, now 10). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 24s Tests passed in hadoop-common.     59m 55s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733678/HADOOP-11973v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0790275 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6736/console This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Committed to trunk, branch-2 and branch-2.7

          Show
          asuresh Arun Suresh added a comment - Committed to trunk, branch-2 and branch-2.7
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7866 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7866/)
          HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7866 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7866/ ) HADOOP-11973 . Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/202/)
          HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/202/ ) HADOOP-11973 . Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #933 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/933/)
          HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #933 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/933/ ) HADOOP-11973 . Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2131 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2131/)
          HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2131 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2131/ ) HADOOP-11973 . Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/191/)
          HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/191/ ) HADOOP-11973 . Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/201/)
          HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/201/ ) HADOOP-11973 . Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2149 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2149/)
          HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2149 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2149/ ) HADOOP-11973 . Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) (Arun Suresh: rev fd3cb533d2495ea220ab2e468835a43a784d7532) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/TestZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java

            People

            • Assignee:
              gchanan Gregory Chanan
              Reporter:
              gchanan Gregory Chanan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development