Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13838

KMSTokenRenewer should close providers

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Critical
        • Resolution: Fixed
        • Affects Version/s: 2.8.0
        • Fix Version/s: 2.8.0, 3.0.0-alpha2
        • Component/s: kms
        • Labels:
          None
        • Target Version/s:
        • Hadoop Flags:
          Reviewed

          Description

          KMSClientProvider need to be closed to free up the SSLFactory internally. See HADOOP-11368 for details.

          Credit to Robert Kanter for finding this.

          1. Text File
            HADOOP-13838.01.patch
            3 kB
            Xiao Chen
          2. Text File
            HADOOP-13838.02.patch
            10 kB
            Xiao Chen

            Issue Links

            relates to

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-13155 Implement TokenRenewer to renew and cancel delegation tokens in KMS

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              xiaochen Xiao Chen added a comment -

              This only applies to the KMSTokenRenwer added by HADOOP-13155. Fixing in patch 1.

              HDFS clients caches the provider in clientcontext, which closes the provider on cache removal.

              Show
              xiaochen Xiao Chen added a comment - This only applies to the KMSTokenRenwer added by HADOOP-13155 . Fixing in patch 1. HDFS clients caches the provider in clientcontext, which closes the provider on cache removal.
              Hide
              Permalink
              rkanter Robert Kanter added a comment -

              The fix seems good to me. Can you add a unit test?

              Show
              rkanter Robert Kanter added a comment - The fix seems good to me. Can you add a unit test?
              Hide
              Permalink
              andrew.wang Andrew Wang added a comment -

              Good catch here Xiao. Do you want to also close the key provider in FSNamesystem? I think this only really affects unit tests, but would be good code hygiene.

              Show
              andrew.wang Andrew Wang added a comment - Good catch here Xiao. Do you want to also close the key provider in FSNamesystem? I think this only really affects unit tests, but would be good code hygiene.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -
              -1 overall



              Vote Subsystem Runtime Comment
              0 reexec 0m 18s Docker mode activated.
              +1 @author 0m 0s The patch does not contain any @author tags.
              -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              +1 mvninstall 8m 46s trunk passed
              +1 compile 10m 1s trunk passed
              +1 checkstyle 0m 28s trunk passed
              +1 mvnsite 1m 4s trunk passed
              +1 mvneclipse 0m 19s trunk passed
              +1 findbugs 1m 34s trunk passed
              +1 javadoc 0m 48s trunk passed
              +1 mvninstall 0m 39s the patch passed
              +1 compile 9m 56s the patch passed
              +1 javac 9m 56s the patch passed
              +1 checkstyle 0m 30s the patch passed
              +1 mvnsite 1m 3s the patch passed
              +1 mvneclipse 0m 21s the patch passed
              +1 whitespace 0m 0s The patch has no whitespace issues.
              +1 findbugs 1m 56s the patch passed
              +1 javadoc 0m 55s the patch passed
              +1 unit 9m 22s hadoop-common in the patch passed.
              +1 asflicense 0m 34s The patch does not generate ASF License warnings.
              50m 32s



              Subsystem Report/Notes
              Docker Image:yetus/hadoop:a9ad5d6
              JIRA Issue HADOOP-13838
              JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840723/HADOOP-13838.01.patch
              Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
              uname Linux c4bbf32e888a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
              Build tool maven
              Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
              git revision trunk / a2b1ff0
              Default Java 1.8.0_111
              findbugs v3.0.0
              Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11150/testReport/
              modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
              Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11150/console
              Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 46s trunk passed +1 compile 10m 1s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 34s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 9m 56s the patch passed +1 javac 9m 56s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 9m 22s hadoop-common in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 50m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840723/HADOOP-13838.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4bbf32e888a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a2b1ff0 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11150/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11150/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
              Hide
              Permalink
              xiaochen Xiao Chen added a comment -

              Thanks a lot for the reviews, Robert and Andrew.

              Patch adds the unit test for KMS, and also closes the instance in FSN.

              Show
              xiaochen Xiao Chen added a comment - Thanks a lot for the reviews, Robert and Andrew. Patch adds the unit test for KMS, and also closes the instance in FSN.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -
              -1 overall



              Vote Subsystem Runtime Comment
              0 reexec 0m 14s Docker mode activated.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              0 mvndep 0m 20s Maven dependency ordering for branch
              +1 mvninstall 6m 55s trunk passed
              +1 compile 9m 31s trunk passed
              +1 checkstyle 1m 37s trunk passed
              +1 mvnsite 2m 25s trunk passed
              +1 mvneclipse 0m 57s trunk passed
              +1 findbugs 3m 46s trunk passed
              +1 javadoc 1m 57s trunk passed
              0 mvndep 0m 17s Maven dependency ordering for patch
              +1 mvninstall 1m 47s the patch passed
              +1 compile 9m 17s the patch passed
              +1 javac 9m 17s the patch passed
              -0 checkstyle 2m 56s root: The patch generated 1 new + 309 unchanged - 0 fixed = 310 total (was 309)
              +1 mvnsite 2m 40s the patch passed
              +1 mvneclipse 1m 6s the patch passed
              +1 whitespace 0m 0s The patch has no whitespace issues.
              +1 findbugs 4m 34s the patch passed
              +1 javadoc 2m 18s the patch passed
              -1 unit 9m 9s hadoop-common in the patch failed.
              +1 unit 2m 26s hadoop-kms in the patch passed.
              +1 unit 70m 49s hadoop-hdfs in the patch passed.
              +1 asflicense 0m 42s The patch does not generate ASF License warnings.
              159m 40s



              Reason Tests
              Failed junit tests hadoop.net.TestDNS



              Subsystem Report/Notes
              Docker Image:yetus/hadoop:a9ad5d6
              JIRA Issue HADOOP-13838
              JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840746/HADOOP-13838.02.patch
              Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
              uname Linux fbc18b9b543d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
              Build tool maven
              Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
              git revision trunk / a2b1ff0
              Default Java 1.8.0_111
              findbugs v3.0.0
              checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/artifact/patchprocess/diff-checkstyle-root.txt
              unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
              Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/testReport/
              modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs U: .
              Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/console
              Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 9m 31s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 2m 25s trunk passed +1 mvneclipse 0m 57s trunk passed +1 findbugs 3m 46s trunk passed +1 javadoc 1m 57s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 47s the patch passed +1 compile 9m 17s the patch passed +1 javac 9m 17s the patch passed -0 checkstyle 2m 56s root: The patch generated 1 new + 309 unchanged - 0 fixed = 310 total (was 309) +1 mvnsite 2m 40s the patch passed +1 mvneclipse 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 34s the patch passed +1 javadoc 2m 18s the patch passed -1 unit 9m 9s hadoop-common in the patch failed. +1 unit 2m 26s hadoop-kms in the patch passed. +1 unit 70m 49s hadoop-hdfs in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 159m 40s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840746/HADOOP-13838.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fbc18b9b543d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a2b1ff0 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11151/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
              Hide
              Permalink
              xiaochen Xiao Chen added a comment -

              Test failure look unrelated, and the checkstyle is about the test method being too long, which was already around the border before this change.

              Show
              xiaochen Xiao Chen added a comment - Test failure look unrelated, and the checkstyle is about the test method being too long, which was already around the border before this change.
              Hide
              Permalink
              rkanter Robert Kanter added a comment -

              +1

              Show
              rkanter Robert Kanter added a comment - +1
              Hide
              Permalink
              rkanter Robert Kanter added a comment -

              Thanks Xiao Chen. Committed to trunk and branch-2!

              Show
              rkanter Robert Kanter added a comment - Thanks Xiao Chen . Committed to trunk and branch-2!
              Hide
              Permalink
              xiaochen Xiao Chen added a comment -

              Thank you Robert Kanter!
              Cherry picked to branch-2.8 as well to match HADOOP-13155.

              Show
              xiaochen Xiao Chen added a comment - Thank you Robert Kanter ! Cherry picked to branch-2.8 as well to match HADOOP-13155 .
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10902 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10902/)
              HADOOP-13838. KMSTokenRenewer should close providers (xiaochen via (rkanter: rev 47ca9e26fba4a639e43bee5bfc001ffc4b42330d)

              • (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
              • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
              • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10902 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10902/ ) HADOOP-13838 . KMSTokenRenewer should close providers (xiaochen via (rkanter: rev 47ca9e26fba4a639e43bee5bfc001ffc4b42330d) (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java

                People

                • Assignee:
                  xiaochen Xiao Chen
                  Reporter:
                  xiaochen Xiao Chen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development