Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10643

Namenode should use loginUser(hdfs) to generateEncryptedKey

    Details

    • Hadoop Flags:
      Reviewed

      Description

      KMSClientProvider is designed to be shared by different KMS clients. When HDFS Namenode as KMS client talks to KMS to generateEncryptedKey for new file creation from proxy user (hive, oozie), the proxyuser handling for KMSClientProvider in this case is unnecessary, which cause 1) an extra proxy user configuration allowing hdfs user to proxy its clients and 2) KMS acls to allow non-hdfs user for GENERATE_EEK operation.

      This ticket is opened to always use HDFS namenode login user (hdfs) when talking to KMS to generateEncryptedKey for new file creation. This way, we have a more secure KMS based HDFS encryption (we can set kms-acls to allow only hdfs user for GENERATE_EEK) with less configuration hassle for KMS to allow hdfs to proxy other users.

      1. HDFS-10643.00.patch
        2 kB
        Xiaoyu Yao
      2. HDFS-10643.01.patch
        2 kB
        Xiaoyu Yao
      3. HDFS-10643.02.patch
        14 kB
        Xiaoyu Yao
      4. HDFS-10643.03.patch
        14 kB
        Xiaoyu Yao
      5. HDFS-10643.04.patch
        14 kB
        Xiaoyu Yao
      6. HDFS-10643.05.patch
        15 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10256/)
          HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. (xyao: rev ec289bbeceff064ad24e189db20a3e0a296822c1)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSecureEncryptionZoneWithKMS.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10256/ ) HDFS-10643 . Namenode should use loginUser(hdfs) to generateEncryptedKey. (xyao: rev ec289bbeceff064ad24e189db20a3e0a296822c1) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSecureEncryptionZoneWithKMS.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java
          Hide
          xyao Xiaoyu Yao added a comment -

          Good catch, Xiao Chen. I'll revert and redo the commit.

          Show
          xyao Xiaoyu Yao added a comment - Good catch, Xiao Chen . I'll revert and redo the commit.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Xiaoyu for the nice work!
          As a note for future jira search, the commit message did not have the jira number HDFS-10643.

          Show
          xiaochen Xiao Chen added a comment - Thanks Xiaoyu for the nice work! As a note for future jira search, the commit message did not have the jira number HDFS-10643 .
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Xiao Chen for the review and Jitendra Nath Pandey for the offline discussion. I've committed the patch to trunk, branch-2 and branch-2.8.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Xiao Chen for the review and Jitendra Nath Pandey for the offline discussion. I've committed the patch to trunk, branch-2 and branch-2.8.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Xiaoyu Yao for explaining and revving! Patch 5 LGTM, +1.

          Show
          xiaochen Xiao Chen added a comment - Thanks Xiaoyu Yao for explaining and revving! Patch 5 LGTM, +1.
          Hide
          xyao Xiaoyu Yao added a comment -

          Open separate ticket HDFS-10748 to track the unit test failure, which is not related to this change.

          Show
          xyao Xiaoyu Yao added a comment - Open separate ticket HDFS-10748 to track the unit test failure, which is not related to this change.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 5s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 1m 3s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 57s the patch passed
          +1 javadoc 0m 51s the patch passed
          -1 unit 78m 22s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          98m 36s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823119/HDFS-10643.05.patch
          JIRA Issue HDFS-10643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 75a2e45c5b75 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89c0bff
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16390/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16390/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16390/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 5s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 57s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 78m 22s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 98m 36s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823119/HDFS-10643.05.patch JIRA Issue HDFS-10643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 75a2e45c5b75 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89c0bff Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16390/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16390/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16390/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Xiao Chen, thanks for the review. This reason for not repro with the unit test is not HDFS-9405. There are 150 key (hadoop.security.kms.client.encrypted.key.cache.size*hadoop.security.kms.client.encrypted.key.cache.low-watermark = 500*0.3=150) getting pre-created upon encryption zone creation. Change the unit test to hadoop.security.kms.client.encrypted.key.cache.size = 4 and hadoop.security.kms.client.encrypted.key.cache.low-watermark=0.5 so that the refill of EDEK cache happen upon the 3rd file creation.

          Update the unit test based on that and now we can repro the original issue without the code fix in the patch with the exception stack below.

          Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: hdfs/localhost@EXAMPLE.COM is not allowed to impersonate oozie_user
          	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
          	at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
          	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:616)
          	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:574)
          	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:91)
          	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:146)
          	at org.apache.hadoop.crypto.key.kms.ValueQueue.getAtMost(ValueQueue.java:299)
          

          Also update the unit test to remove the try/catch as suggested. Please review and let me know your thoughts. Thanks!

          Show
          xyao Xiaoyu Yao added a comment - Xiao Chen , thanks for the review. This reason for not repro with the unit test is not HDFS-9405 . There are 150 key (hadoop.security.kms.client.encrypted.key.cache.size*hadoop.security.kms.client.encrypted.key.cache.low-watermark = 500*0.3=150) getting pre-created upon encryption zone creation. Change the unit test to hadoop.security.kms.client.encrypted.key.cache.size = 4 and hadoop.security.kms.client.encrypted.key.cache.low-watermark=0.5 so that the refill of EDEK cache happen upon the 3rd file creation. Update the unit test based on that and now we can repro the original issue without the code fix in the patch with the exception stack below. Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: hdfs/localhost@EXAMPLE.COM is not allowed to impersonate oozie_user at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:616) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:574) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$200(KMSClientProvider.java:91) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:146) at org.apache.hadoop.crypto.key.kms.ValueQueue.getAtMost(ValueQueue.java:299) Also update the unit test to remove the try/catch as suggested. Please review and let me know your thoughts. Thanks!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Xiaoyu Yao for revving!

          The change LGTM too, but the test is passing even without the fix. I think (not debugged, sorry if not correct) this is because NN will warm up the cache after HDFS-9405, so the test didn't trigger the KMS ACL check.
          Why createFile is done 3 times in the test? Is it for cache draining? I think we could set the cache size to 1 make it fail if so.

          Also a nit: in the test, can we remove this?

          try {
          ...
          } catch (IOException e) {
              throw new IOException(e);
          }
          
          Show
          xiaochen Xiao Chen added a comment - Thanks Xiaoyu Yao for revving! The change LGTM too, but the test is passing even without the fix. I think (not debugged, sorry if not correct) this is because NN will warm up the cache after HDFS-9405 , so the test didn't trigger the KMS ACL check. Why createFile is done 3 times in the test? Is it for cache draining? I think we could set the cache size to 1 make it fail if so. Also a nit: in the test, can we remove this? try { ... } catch (IOException e) { throw new IOException(e); }
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 3s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 59m 58s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          79m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821442/HDFS-10643.04.patch
          JIRA Issue HDFS-10643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6201e31dde22 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9f473cf
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16280/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16280/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 3s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 59m 58s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821442/HDFS-10643.04.patch JIRA Issue HDFS-10643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6201e31dde22 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f473cf Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16280/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16280/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          +1

          Show
          jnp Jitendra Nath Pandey added a comment - +1
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Jitendra Nath Pandey for the review. Attach a patch that address the comments.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Jitendra Nath Pandey for the review. Attach a patch that address the comments.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          Minor comment:
          The edek declaration and assignment could be done on the same line i.e.

          EncryptedKeyVersion edek = SecurityUtil.doAs....
          
          Show
          jnp Jitendra Nath Pandey added a comment - Minor comment: The edek declaration and assignment could be done on the same line i.e. EncryptedKeyVersion edek = SecurityUtil.doAs....
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 55s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 22s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 75m 31s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          94m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821162/HDFS-10643.03.patch
          JIRA Issue HDFS-10643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3095510491da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ce93595
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16267/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16267/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 55s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 75m 31s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 94m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821162/HDFS-10643.03.patch JIRA Issue HDFS-10643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3095510491da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ce93595 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16267/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16267/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Fix the checkstyle issue.

          Show
          xyao Xiaoyu Yao added a comment - Fix the checkstyle issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 11s trunk passed
          +1 compile 0m 56s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 58m 21s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          79m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821115/HDFS-10643.02.patch
          JIRA Issue HDFS-10643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0635a7f3c906 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 95f2b98
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16263/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16263/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16263/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 11s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 58m 21s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 79m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821115/HDFS-10643.02.patch JIRA Issue HDFS-10643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0635a7f3c906 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 95f2b98 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16263/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16263/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16263/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a patch with unit test that use kerby based minikdc, minikms and minidfscluster.

          Show
          xyao Xiaoyu Yao added a comment - Attach a patch with unit test that use kerby based minikdc, minikms and minidfscluster.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 54s the patch passed
          +1 unit 58m 21s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          78m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818706/HDFS-10643.01.patch
          JIRA Issue HDFS-10643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9d10856cefc2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 92fe2db
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16084/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16084/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16084/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 7s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 54s the patch passed +1 unit 58m 21s hadoop-hdfs in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 78m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818706/HDFS-10643.01.patch JIRA Issue HDFS-10643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9d10856cefc2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 92fe2db Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16084/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16084/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16084/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Xiaoyu Yao for opening the issue and the patch.
          I think the idea makes sense, since from HDFS perspective the only user needs to generate EDEK is hdfs. Ping Andrew Wang for awareness.

          Regarding checkTGTAndReloginFromKeytab, you're absolutely right that we don't need it in the client code here. I think adding it to KerberosAuthencitator makes sense logically, and in that case we don't need these in DTA any more.

            public void authenticate(URL url, AuthenticatedURL.Token token)
                throws IOException, AuthenticationException {
              if (!hasDelegationToken(url, token)) {
                // check and renew TGT to handle potential expiration
                UserGroupInformation.getCurrentUser().checkTGTAndReloginFromKeytab();
                authenticator.authenticate(url, token);
              }
            }
          

          I didn't put it there in HADOOP-13255 because KA is in hadoop-auth component, while DTA and UGI are both in hadoop-common. Feels like we'll need a dependency between the two in order to add this... Let's follow up on this in the separate jira.

          Show
          xiaochen Xiao Chen added a comment - Thanks Xiaoyu Yao for opening the issue and the patch. I think the idea makes sense, since from HDFS perspective the only user needs to generate EDEK is hdfs . Ping Andrew Wang for awareness. Regarding checkTGTAndReloginFromKeytab , you're absolutely right that we don't need it in the client code here. I think adding it to KerberosAuthencitator makes sense logically, and in that case we don't need these in DTA any more. public void authenticate(URL url, AuthenticatedURL.Token token) throws IOException, AuthenticationException { if (!hasDelegationToken(url, token)) { // check and renew TGT to handle potential expiration UserGroupInformation.getCurrentUser().checkTGTAndReloginFromKeytab(); authenticator.authenticate(url, token); } } I didn't put it there in HADOOP-13255 because KA is in hadoop-auth component, while DTA and UGI are both in hadoop-common. Feels like we'll need a dependency between the two in order to add this... Let's follow up on this in the separate jira.
          Hide
          xyao Xiaoyu Yao added a comment -

          checkTGTAndReloginFromKeytab is not needed with HADOOP-13255 per discussion with Jitendra Nath Pandey. Adding a patch v1 for that.
          I'm working on the unit test of this and will update the patch again later.

          I also found a potential issue with HADOOP-13255 where the checkTGTAndReloginFromKeytab is invoked with only DelegationTokenAuthenticator#authenticate but not KerberosAuthenticator#authenticate. This is not an issue now because we currently don't use KerberosAuthenticator directly. Only DelegationTokenAuthenticator or KerberosDelegationTokenAuthenticator are being used. Since both KerberosAuthenticator and DelegationTokenAuthenticator implement the Authenticator interface, it is good to have checkTGTAndReloginFromKeytab added to authenticate implementations for consistency. I will open a separate ticket for it.

          cc: Xiao Chen and Zhe Zhang for additional feedback.

          Show
          xyao Xiaoyu Yao added a comment - checkTGTAndReloginFromKeytab is not needed with HADOOP-13255 per discussion with Jitendra Nath Pandey . Adding a patch v1 for that. I'm working on the unit test of this and will update the patch again later. I also found a potential issue with HADOOP-13255 where the checkTGTAndReloginFromKeytab is invoked with only DelegationTokenAuthenticator#authenticate but not KerberosAuthenticator#authenticate . This is not an issue now because we currently don't use KerberosAuthenticator directly. Only DelegationTokenAuthenticator or KerberosDelegationTokenAuthenticator are being used. Since both KerberosAuthenticator and DelegationTokenAuthenticator implement the Authenticator interface, it is good to have checkTGTAndReloginFromKeytab added to authenticate implementations for consistency. I will open a separate ticket for it. cc: Xiao Chen and Zhe Zhang for additional feedback.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 2s trunk passed
          +1 compile 0m 50s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 59s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 50s the patch passed
          +1 javac 0m 50s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 57s the patch passed
          -1 unit 78m 27s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          99m 6s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818429/HDFS-10643.00.patch
          JIRA Issue HDFS-10643
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ab1aa255d734 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5b4a708
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16078/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16078/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 2s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 59s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 50s the patch passed +1 javac 0m 50s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 78m 27s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 99m 6s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818429/HDFS-10643.00.patch JIRA Issue HDFS-10643 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ab1aa255d734 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5b4a708 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16078/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16078/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a patch to illustrate the idea. Will add unit test later.

          Show
          xyao Xiaoyu Yao added a comment - Attach a patch to illustrate the idea. Will add unit test later.

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development