Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10918

Add a tool to get FileEncryptionInfo from CLI

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. HDFS-10918.01.patch
      11 kB
      Xiao Chen
    2. HDFS-10918.02.patch
      12 kB
      Xiao Chen
    3. HDFS-10918.03.patch
      13 kB
      Xiao Chen
    4. HDFS-10918.04.patch
      14 kB
      Xiao Chen
    5. HDFS-10918.05.patch
      14 kB
      Xiao Chen

      Activity

      Hide
      yzhangal Yongjun Zhang added a comment -

      Hi Manikandan R and Xiao Chen,

      I did run into the same failure, I did do "mvn clean" but for some reason the old HdfsAdmin.class file was not cleaned up. I tried quite a few times and the problem remains. A later attempt did clean it up though.

      Thanks.

      Show
      yzhangal Yongjun Zhang added a comment - Hi Manikandan R and Xiao Chen , I did run into the same failure, I did do "mvn clean" but for some reason the old HdfsAdmin.class file was not cleaned up. I tried quite a few times and the problem remains. A later attempt did clean it up though. Thanks.
      Hide
      manirajv06@gmail.com Manikandan R added a comment -

      Hi Xiao Chen,

      Looks like my earlier mvn clean also didn't went through completely. Tried again now and able to build it. Thanks.

      Show
      manirajv06@gmail.com Manikandan R added a comment - Hi Xiao Chen , Looks like my earlier mvn clean also didn't went through completely. Tried again now and able to build it. Thanks.
      Hide
      xiaochen Xiao Chen added a comment -

      Hi Manikandan R,
      Thanks for reporting this. Could you share which exact commit did you reproduce this? This is committed in early October and I have not seen this.

      Yongjun Zhang had a similar error earlier, and it seems somehow mvn clean on his local wasn't complete. Yongjun, does this ring any bells to you? Thanks.

      Show
      xiaochen Xiao Chen added a comment - Hi Manikandan R , Thanks for reporting this. Could you share which exact commit did you reproduce this? This is committed in early October and I have not seen this. Yongjun Zhang had a similar error earlier, and it seems somehow mvn clean on his local wasn't complete. Yongjun, does this ring any bells to you? Thanks.
      Hide
      manirajv06@gmail.com Manikandan R added a comment -

      I recently pulled code from git and facing maven build errors. Please see below.

      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
      [ERROR] /home/mani/codebase/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CryptoAdmin.java:[230,18] cannot find symbol
      [ERROR] symbol: method getFileEncryptionInfo(org.apache.hadoop.fs.Path)
      [ERROR] location: variable admin of type org.apache.hadoop.hdfs.client.HdfsAdmin
      [ERROR] -> [Help 1]

      Show
      manirajv06@gmail.com Manikandan R added a comment - I recently pulled code from git and facing maven build errors. Please see below. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-hdfs: Compilation failure [ERROR] /home/mani/codebase/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CryptoAdmin.java: [230,18] cannot find symbol [ERROR] symbol: method getFileEncryptionInfo(org.apache.hadoop.fs.Path) [ERROR] location: variable admin of type org.apache.hadoop.hdfs.client.HdfsAdmin [ERROR] -> [Help 1]
      Hide
      hudson Hudson added a comment -

      SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10535 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10535/)
      HDFS-10918. Add a tool to get FileEncryptionInfo from CLI. Contributed (xiao: rev 853d65a157362661ccab10379c2d82e780382f83)

      • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md
      • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileEncryptionInfo.java
      • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CryptoAdmin.java
      • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCryptoConf.xml
      • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java
      • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
      Show
      hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10535 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10535/ ) HDFS-10918 . Add a tool to get FileEncryptionInfo from CLI. Contributed (xiao: rev 853d65a157362661ccab10379c2d82e780382f83) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/TransparentEncryption.md (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileEncryptionInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CryptoAdmin.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testCryptoConf.xml (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
      Hide
      xiaochen Xiao Chen added a comment -

      Committed this to trunk, branch-2 and branch-2.8.
      Thanks a lot Andrew Wang for the great reviews and patient explanations!

      Show
      xiaochen Xiao Chen added a comment - Committed this to trunk, branch-2 and branch-2.8. Thanks a lot Andrew Wang for the great reviews and patient explanations!
      Hide
      xiaochen Xiao Chen added a comment -

      Failed tests look unrelated and passed locally.
      Will wait for some earth rotations in case anyone has interests, plan to commit this on Monday.

      Show
      xiaochen Xiao Chen added a comment - Failed tests look unrelated and passed locally. Will wait for some earth rotations in case anyone has interests, plan to commit this on Monday.
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 16s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
      0 mvndep 1m 43s Maven dependency ordering for branch
      +1 mvninstall 8m 30s trunk passed
      +1 compile 7m 21s trunk passed
      +1 checkstyle 1m 30s trunk passed
      +1 mvnsite 2m 28s trunk passed
      +1 mvneclipse 0m 40s trunk passed
      +1 findbugs 4m 28s trunk passed
      +1 javadoc 2m 2s trunk passed
      0 mvndep 0m 14s Maven dependency ordering for patch
      +1 mvninstall 2m 2s the patch passed
      +1 compile 6m 49s the patch passed
      +1 javac 6m 49s the patch passed
      +1 checkstyle 1m 25s the patch passed
      +1 mvnsite 2m 21s the patch passed
      +1 mvneclipse 0m 37s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 xml 0m 2s The patch has no ill-formed XML file.
      +1 findbugs 4m 56s the patch passed
      +1 javadoc 2m 1s the patch passed
      -1 unit 7m 54s hadoop-common in the patch failed.
      +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
      -1 unit 58m 24s hadoop-hdfs in the patch failed.
      +1 asflicense 0m 25s The patch does not generate ASF License warnings.
      118m 6s



      Reason Tests
      Failed junit tests hadoop.net.TestDNS
        hadoop.hdfs.TestDFSShell
        hadoop.hdfs.TestRenameWhileOpen



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Issue HDFS-10918
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831129/HDFS-10918.05.patch
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
      uname Linux 4443968100ff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / 0670149
      Default Java 1.8.0_101
      findbugs v3.0.0
      unit https://builds.apache.org/job/PreCommit-HDFS-Build/16947/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
      unit https://builds.apache.org/job/PreCommit-HDFS-Build/16947/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
      Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16947/testReport/
      modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
      Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16947/console
      Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 43s Maven dependency ordering for branch +1 mvninstall 8m 30s trunk passed +1 compile 7m 21s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 4m 28s trunk passed +1 javadoc 2m 2s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 6m 49s the patch passed +1 javac 6m 49s the patch passed +1 checkstyle 1m 25s the patch passed +1 mvnsite 2m 21s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 56s the patch passed +1 javadoc 2m 1s the patch passed -1 unit 7m 54s hadoop-common in the patch failed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 58m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 118m 6s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.TestDFSShell   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10918 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831129/HDFS-10918.05.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4443968100ff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0670149 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16947/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16947/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16947/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16947/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
      Hide
      xiaochen Xiao Chen added a comment -

      Oops, new test case caught the command renaming..

      Show
      xiaochen Xiao Chen added a comment - Oops, new test case caught the command renaming..
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 19s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
      0 mvndep 1m 37s Maven dependency ordering for branch
      +1 mvninstall 7m 3s trunk passed
      +1 compile 7m 20s trunk passed
      +1 checkstyle 1m 29s trunk passed
      +1 mvnsite 2m 25s trunk passed
      +1 mvneclipse 0m 39s trunk passed
      +1 findbugs 4m 41s trunk passed
      +1 javadoc 2m 4s trunk passed
      0 mvndep 0m 14s Maven dependency ordering for patch
      +1 mvninstall 2m 8s the patch passed
      +1 compile 7m 40s the patch passed
      +1 javac 7m 40s the patch passed
      +1 checkstyle 1m 32s the patch passed
      +1 mvnsite 2m 35s the patch passed
      +1 mvneclipse 0m 38s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 xml 0m 1s The patch has no ill-formed XML file.
      +1 findbugs 5m 12s the patch passed
      +1 javadoc 2m 4s the patch passed
      +1 unit 8m 12s hadoop-common in the patch passed.
      +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
      -1 unit 61m 47s hadoop-hdfs in the patch failed.
      +1 asflicense 0m 24s The patch does not generate ASF License warnings.
      122m 4s



      Reason Tests
      Failed junit tests hadoop.cli.TestCryptoAdminCLI



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Issue HDFS-10918
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831025/HDFS-10918.04.patch
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
      uname Linux 39562a53928e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / 10be459
      Default Java 1.8.0_101
      findbugs v3.0.0
      unit https://builds.apache.org/job/PreCommit-HDFS-Build/16937/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
      Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16937/testReport/
      modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
      Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16937/console
      Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 37s Maven dependency ordering for branch +1 mvninstall 7m 3s trunk passed +1 compile 7m 20s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 2m 25s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 4m 41s trunk passed +1 javadoc 2m 4s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 8s the patch passed +1 compile 7m 40s the patch passed +1 javac 7m 40s the patch passed +1 checkstyle 1m 32s the patch passed +1 mvnsite 2m 35s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 12s the patch passed +1 javadoc 2m 4s the patch passed +1 unit 8m 12s hadoop-common in the patch passed. +1 unit 0m 58s hadoop-hdfs-client in the patch passed. -1 unit 61m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 122m 4s Reason Tests Failed junit tests hadoop.cli.TestCryptoAdminCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10918 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831025/HDFS-10918.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 39562a53928e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 10be459 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16937/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16937/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16937/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
      Hide
      xiaochen Xiao Chen added a comment - - edited

      Patch 4 to true up the docs.

      Show
      xiaochen Xiao Chen added a comment - - edited Patch 4 to true up the docs.
      Hide
      andrew.wang Andrew Wang added a comment -

      Great, this looks right. Some last little nits:

      • In the doc, we should also add a table with the arguments like the other commands.
      • With just this help text, it's not clear what info is returned by this command and why a user might care. Consider expanding the help text about usecases, and adding sample output to the doc.

      +1 pending though, looks good overall.

      Show
      andrew.wang Andrew Wang added a comment - Great, this looks right. Some last little nits: In the doc, we should also add a table with the arguments like the other commands. With just this help text, it's not clear what info is returned by this command and why a user might care. Consider expanding the help text about usecases, and adding sample output to the doc. +1 pending though, looks good overall.
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 16s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
      0 mvndep 0m 14s Maven dependency ordering for branch
      +1 mvninstall 6m 48s trunk passed
      +1 compile 6m 47s trunk passed
      +1 checkstyle 1m 26s trunk passed
      +1 mvnsite 2m 21s trunk passed
      +1 mvneclipse 0m 37s trunk passed
      +1 findbugs 4m 23s trunk passed
      +1 javadoc 2m 0s trunk passed
      0 mvndep 0m 14s Maven dependency ordering for patch
      +1 mvninstall 1m 52s the patch passed
      +1 compile 6m 45s the patch passed
      +1 javac 6m 45s the patch passed
      +1 checkstyle 1m 26s the patch passed
      +1 mvnsite 2m 17s the patch passed
      +1 mvneclipse 0m 37s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 xml 0m 1s The patch has no ill-formed XML file.
      +1 findbugs 4m 45s the patch passed
      +1 javadoc 2m 2s the patch passed
      +1 unit 7m 43s hadoop-common in the patch passed.
      +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
      -1 unit 60m 16s hadoop-hdfs in the patch failed.
      +1 asflicense 0m 24s The patch does not generate ASF License warnings.
      115m 7s



      Reason Tests
      Failed junit tests hadoop.cli.TestCryptoAdminCLI
        hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
        hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Issue HDFS-10918
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830831/HDFS-10918.03.patch
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
      uname Linux 9f910d43f786 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / 47f8092
      Default Java 1.8.0_101
      findbugs v3.0.0
      unit https://builds.apache.org/job/PreCommit-HDFS-Build/16919/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
      Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16919/testReport/
      modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
      Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16919/console
      Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 48s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 2m 21s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 4m 23s trunk passed +1 javadoc 2m 0s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 6m 45s the patch passed +1 javac 6m 45s the patch passed +1 checkstyle 1m 26s the patch passed +1 mvnsite 2m 17s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 45s the patch passed +1 javadoc 2m 2s the patch passed +1 unit 7m 43s hadoop-common in the patch passed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 60m 16s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 115m 7s Reason Tests Failed junit tests hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10918 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830831/HDFS-10918.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 9f910d43f786 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 47f8092 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16919/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16919/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16919/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
      Hide
      xiaochen Xiao Chen added a comment -

      I see.... sorry that was a stupid question, thanks a lot for explaining to me.
      Patch 3 attached to fix that, and rebase against latest trunk.

      Show
      xiaochen Xiao Chen added a comment - I see.... sorry that was a stupid question, thanks a lot for explaining to me. Patch 3 attached to fix that, and rebase against latest trunk.
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 0s Docker mode activated.
      -1 patch 0m 8s HDFS-10918 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



      Subsystem Report/Notes
      JIRA Issue HDFS-10918
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830817/HDFS-10918.02.patch
      Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16916/console
      Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 8s HDFS-10918 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10918 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830817/HDFS-10918.02.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16916/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
      Hide
      andrew.wang Andrew Wang added a comment -

      I think we still are missing handling relative paths and symlinks. Let's look at getFileStatus:

        public FileStatus getFileStatus(Path f) throws IOException {
          Path absF = fixRelativePart(f); // <----------- uses the CWD to turn relative into absolute paths
          return new FileSystemLinkResolver<FileStatus>() { // <------- class to keep following symlinks until path is fully resolved
            @Override
            public FileStatus doCall(final Path p) throws IOException {
              HdfsFileStatus fi = dfs.getFileInfo(getPathName(p));
              if (fi != null) {
                return fi.makeQualified(getUri(), p);
              } else {
                throw new FileNotFoundException("File does not exist: " + p);
              }
            }
            @Override
            public FileStatus next(final FileSystem fs, final Path p)
                throws IOException {
              return fs.getFileStatus(p);
            }
          }.resolve(this, absF);
        }
      

      Looking at this more closely myself, code sharing is pretty tough because we need that HdfsFileStatus which is only returned by an HDFS. There are other examples of ops that can only resolve symlinks through a DFS instance, e.g. renameSnapshot, that you can use as an example.

      Show
      andrew.wang Andrew Wang added a comment - I think we still are missing handling relative paths and symlinks. Let's look at getFileStatus: public FileStatus getFileStatus(Path f) throws IOException { Path absF = fixRelativePart(f); // <----------- uses the CWD to turn relative into absolute paths return new FileSystemLinkResolver<FileStatus>() { // <------- class to keep following symlinks until path is fully resolved @Override public FileStatus doCall( final Path p) throws IOException { HdfsFileStatus fi = dfs.getFileInfo(getPathName(p)); if (fi != null ) { return fi.makeQualified(getUri(), p); } else { throw new FileNotFoundException( "File does not exist: " + p); } } @Override public FileStatus next( final FileSystem fs, final Path p) throws IOException { return fs.getFileStatus(p); } }.resolve( this , absF); } Looking at this more closely myself, code sharing is pretty tough because we need that HdfsFileStatus which is only returned by an HDFS. There are other examples of ops that can only resolve symlinks through a DFS instance, e.g. renameSnapshot , that you can use as an example.
      Hide
      xiaochen Xiao Chen added a comment -

      Thanks for the review and the interesting offline chat Andrew.

      Patch 2 addresses all comments (good catch) except #2:
      Can't 'cast' to HdfsFileStatus since it's a separate class. So didn't change any more towards code sharing.

      Symlinks also.

      I think this handles symlink, since DFSClient will call with getFileInfo(src, true).

      Show
      xiaochen Xiao Chen added a comment - Thanks for the review and the interesting offline chat Andrew. Patch 2 addresses all comments (good catch) except #2: Can't 'cast' to HdfsFileStatus since it's a separate class. So didn't change any more towards code sharing. Symlinks also. I think this handles symlink, since DFSClient will call with getFileInfo(src, true) .
      Hide
      andrew.wang Andrew Wang added a comment -

      Thanks for working on this Xiao, a few comments:

      • I don't think we need that checkAccess call, since the NN will enforce permissions already when the client calls getFileInfo
      • Is it possible to call dfs.getFileStatus and cast to a HdfsFileStatus, or some other way of code sharing? Note that there's extra handling we need to do to handle relative paths which is not handled by the current getFeInfo implementation. Symlinks also.
      • Rather than "getFeInfo" in CryptoAdmin, can we expand to "getFileEncryptionInfo" for the user visible flag?
      • This probably also needs a doc update
      Show
      andrew.wang Andrew Wang added a comment - Thanks for working on this Xiao, a few comments: I don't think we need that checkAccess call, since the NN will enforce permissions already when the client calls getFileInfo Is it possible to call dfs.getFileStatus and cast to a HdfsFileStatus, or some other way of code sharing? Note that there's extra handling we need to do to handle relative paths which is not handled by the current getFeInfo implementation. Symlinks also. Rather than "getFeInfo" in CryptoAdmin, can we expand to "getFileEncryptionInfo" for the user visible flag? This probably also needs a doc update
      Hide
      xiaochen Xiao Chen added a comment -

      Hi Andrew Wang,
      Could you take a look and see if this makes sense to you? Thanks a lot.

      Show
      xiaochen Xiao Chen added a comment - Hi Andrew Wang , Could you take a look and see if this makes sense to you? Thanks a lot.
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      0 reexec 0m 14s Docker mode activated.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
      0 mvndep 0m 14s Maven dependency ordering for branch
      +1 mvninstall 6m 51s trunk passed
      +1 compile 7m 1s trunk passed
      +1 checkstyle 1m 28s trunk passed
      +1 mvnsite 2m 21s trunk passed
      +1 mvneclipse 0m 37s trunk passed
      +1 findbugs 4m 23s trunk passed
      +1 javadoc 2m 2s trunk passed
      0 mvndep 0m 14s Maven dependency ordering for patch
      +1 mvninstall 1m 59s the patch passed
      +1 compile 7m 4s the patch passed
      +1 javac 7m 4s the patch passed
      +1 checkstyle 1m 29s the patch passed
      +1 mvnsite 2m 21s the patch passed
      +1 mvneclipse 0m 38s the patch passed
      +1 whitespace 0m 0s The patch has no whitespace issues.
      +1 xml 0m 1s The patch has no ill-formed XML file.
      +1 findbugs 4m 58s the patch passed
      +1 javadoc 2m 7s the patch passed
      +1 unit 7m 22s hadoop-common in the patch passed.
      +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
      -1 unit 63m 22s hadoop-hdfs in the patch failed.
      +1 asflicense 0m 24s The patch does not generate ASF License warnings.
      119m 5s



      Reason Tests
      Failed junit tests hadoop.hdfs.server.mover.TestStorageMover
        hadoop.hdfs.server.blockmanagement.TestBlockManager



      Subsystem Report/Notes
      Docker Image:yetus/hadoop:9560f25
      JIRA Issue HDFS-10918
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830616/HDFS-10918.01.patch
      Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
      uname Linux b5ae09d8ed2a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
      git revision trunk / d144398
      Default Java 1.8.0_101
      findbugs v3.0.0
      unit https://builds.apache.org/job/PreCommit-HDFS-Build/16895/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
      Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16895/testReport/
      modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
      Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16895/console
      Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 7m 1s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 2m 21s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 4m 23s trunk passed +1 javadoc 2m 2s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 7m 4s the patch passed +1 javac 7m 4s the patch passed +1 checkstyle 1m 29s the patch passed +1 mvnsite 2m 21s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 58s the patch passed +1 javadoc 2m 7s the patch passed +1 unit 7m 22s hadoop-common in the patch passed. +1 unit 0m 56s hadoop-hdfs-client in the patch passed. -1 unit 63m 22s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 119m 5s Reason Tests Failed junit tests hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.blockmanagement.TestBlockManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10918 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830616/HDFS-10918.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux b5ae09d8ed2a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d144398 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16895/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16895/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16895/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
      Hide
      xiaochen Xiao Chen added a comment -

      Attaching a patch to express the idea.

      Some explanations:

      • Client can already get FileEncryptionInfo from DFSClient code, just not convenient from CLI. So adding such a tool provides some convenience, with no security drawback.
      • Anyone can read the file should be able to get feinfo, to be consistent with client behavior.
      • Added a toStringStable to feinfo, similar to what HDFS-9732 is done.
      • Can't seem to have permission related tests added to TestCryptoAdminCLI, since it's always running as superuser. Manually tested though.
      Show
      xiaochen Xiao Chen added a comment - Attaching a patch to express the idea. Some explanations: Client can already get FileEncryptionInfo from DFSClient code, just not convenient from CLI. So adding such a tool provides some convenience, with no security drawback. Anyone can read the file should be able to get feinfo, to be consistent with client behavior. Added a toStringStable to feinfo, similar to what HDFS-9732 is done. Can't seem to have permission related tests added to TestCryptoAdminCLI , since it's always running as superuser. Manually tested though.

        People

        • Assignee:
          xiaochen Xiao Chen
          Reporter:
          xiaochen Xiao Chen
        • Votes:
          0 Vote for this issue
          Watchers:
          7 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development