Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12659

Incorrect usage of config parameters in token manager of KMS

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1, 2.6.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hi, the usage of the following configs of Key Management Server (KMS) are problematic:
      hadoop.kms.authentication.delegation-token.renew-interval.sec
      hadoop.kms.authentication.delegation-token.removal-scan-interval.sec

      The name indicates that the units are sec, and the online doc shows that the default values are 86400 and 3600, respectively.
      https://hadoop.apache.org/docs/stable/hadoop-kms/index.html
      which is also defined in

      DelegationTokenManager.java
       55   public static final String RENEW_INTERVAL = PREFIX + "renew-interval.sec";
       56   public static final long RENEW_INTERVAL_DEFAULT = 24 * 60 * 60;
       ...
       58   public static final String REMOVAL_SCAN_INTERVAL = PREFIX +
       59       "removal-scan-interval.sec";
       60   public static final long REMOVAL_SCAN_INTERVAL_DEFAULT = 60 * 60;
      

      However, in DelegationTokenManager.java and ZKDelegationTokenSecretManager.java, these two parameters are used incorrectly.

      1. DelegationTokenManager.java

       70           conf.getLong(RENEW_INTERVAL, RENEW_INTERVAL_DEFAULT) * 1000,
       71           conf.getLong(REMOVAL_SCAN_INTERVAL, 
       72               REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
      

      Apparently, at Line 72, REMOVAL_SCAN_INTERVAL should be used in the same way as RENEW_INTERVAL, like

      72c72
      <               REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
      ---
      >               REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
      

      Currently, the unit of hadoop.kms.authentication.delegation-token.removal-scan-interval.sec is not sec but millisec.

      2. ZKDelegationTokenSecretManager.java

      142         conf.getLong(DelegationTokenManager.RENEW_INTERVAL,
      143             DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
      144         conf.getLong(DelegationTokenManager.REMOVAL_SCAN_INTERVAL,
      145             DelegationTokenManager.REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
      

      The situation is the opposite in this class that hadoop.kms.authentication.delegation-token.renew-interval.sec is wrong but the other is correct...
      A patch should be like

      143c143
      <             DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
      ---
      >             DelegationTokenManager.RENEW_INTERVAL_DEFAULT) * 1000,
      

      Thanks!

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9157 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9157/)
          HADOOP-12659. Incorrect usage of config parameters in token manager of (xyao: rev f3427d3766d7101d0d1c37d6281918551d221ebe)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9157 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9157/ ) HADOOP-12659 . Incorrect usage of config parameters in token manager of (xyao: rev f3427d3766d7101d0d1c37d6281918551d221ebe) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you Arun Suresh and Xiaoyu Yao for the review.

          Show
          liuml07 Mingliang Liu added a comment - Thank you Arun Suresh and Xiaoyu Yao for the review.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Mingliang Liu for the contribution and all for the reviews. I've committed the patch to trunk, branch-2 and branch-2.8.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Mingliang Liu for the contribution and all for the reviews. I've committed the patch to trunk, branch-2 and branch-2.8.
          Hide
          xyao Xiaoyu Yao added a comment -

          Patch look good to me. +1. I will commit it shortly.

          Show
          xyao Xiaoyu Yao added a comment - Patch look good to me. +1. I will commit it shortly.
          Hide
          tianyin Tianyin Xu added a comment -
          Show
          tianyin Tianyin Xu added a comment - Thanks, Mingliang Liu and Arun Suresh !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 13s trunk passed
          +1 compile 10m 34s trunk passed with JDK v1.8.0_66
          +1 compile 9m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 11s trunk passed
          +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 6s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 38s the patch passed
          +1 compile 8m 44s the patch passed with JDK v1.8.0_66
          +1 javac 8m 44s the patch passed
          +1 compile 9m 25s the patch passed with JDK v1.7.0_91
          +1 javac 9m 25s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 6s the patch passed with JDK v1.7.0_91
          +1 unit 8m 24s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.7.0_91.
          -1 asflicense 0m 23s Patch generated 1 ASF License warnings.
          79m 17s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778656/HADOOP-12659.000.patch
          JIRA Issue HADOOP-12659
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 41d4dea66e78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8652cce
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8282/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8282/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8282/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 13s trunk passed +1 compile 10m 34s trunk passed with JDK v1.8.0_66 +1 compile 9m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 11s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 6s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 38s the patch passed +1 compile 8m 44s the patch passed with JDK v1.8.0_66 +1 javac 8m 44s the patch passed +1 compile 9m 25s the patch passed with JDK v1.7.0_91 +1 javac 9m 25s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 6s the patch passed with JDK v1.7.0_91 +1 unit 8m 24s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 23s Patch generated 1 ASF License warnings. 79m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778656/HADOOP-12659.000.patch JIRA Issue HADOOP-12659 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 41d4dea66e78 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8652cce findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8282/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8282/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8282/console This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for reporting this... The patch looks good.
          +1 pending jenkins

          Show
          asuresh Arun Suresh added a comment - Thanks for reporting this... The patch looks good. +1 pending jenkins
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Tianyin Xu for reporting this. It is a very nice catch. The v0 patch is to address this.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Tianyin Xu for reporting this. It is a very nice catch. The v0 patch is to address this.

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              tianyin Tianyin Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development