Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.1
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: datanode, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      StorageTypes are now encoded in the BlockTokenIdentifier to ensure that the intended StorageType for writes is not tampered with on it's way through the Client to the Datanode.

      Description

      HDFS-6702 is adding support for file creation based on StorageType.

      The block token is used as a tamper-proof channel for communicating block parameters from the NN to the DN during block creation. The StorageType should be included in this block token.

      1. HDFS-6708.0001.patch
        43 kB
        Ewan Higgs
      2. HDFS-6708.0002.patch
        43 kB
        Ewan Higgs
      3. HDFS-6708.0003.patch
        51 kB
        Ewan Higgs
      4. HDFS-6708.0004.patch
        52 kB
        Ewan Higgs
      5. HDFS-6708.0005.patch
        52 kB
        Ewan Higgs
      6. HDFS-6708.0006.patch
        52 kB
        Ewan Higgs
      7. HDFS-6708.0007.patch
        53 kB
        Ewan Higgs
      8. HDFS-6708.0008.patch
        49 kB
        Chris Douglas
      9. HDFS-6708.0009.patch
        48 kB
        Chris Douglas
      10. HDFS-6708.0010.patch
        51 kB
        Chris Douglas

        Issue Links

          Activity

          Hide
          chris.douglas Chris Douglas added a comment -

          Forgot to mention that I ran Yetus and the HDFS tests on my dev box before committing.

          Show
          chris.douglas Chris Douglas added a comment - Forgot to mention that I ran Yetus and the HDFS tests on my dev box before committing.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11633 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11633/)
          HDFS-6708. StorageType should be encoded in the block token. Contributed (cdouglas: rev 2f73396b5901fd5fe29f6cd76fc1b3134b854b37)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/KeyManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockPoolTokenSecretManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/token/block/TestBlockToken.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockWriter.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11633 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11633/ ) HDFS-6708 . StorageType should be encoded in the block token. Contributed (cdouglas: rev 2f73396b5901fd5fe29f6cd76fc1b3134b854b37) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/KeyManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockPoolTokenSecretManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/token/block/TestBlockToken.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockWriter.java
          Hide
          chris.douglas Chris Douglas added a comment -

          I committed this. Thanks Ewan

          Show
          chris.douglas Chris Douglas added a comment - I committed this. Thanks Ewan
          Hide
          chris.douglas Chris Douglas added a comment -

          Ewan Higgs, you're right, that's not going to work with HDFS-9807. Changed it back to StorageType[]

          Show
          chris.douglas Chris Douglas added a comment - Ewan Higgs , you're right, that's not going to work with HDFS-9807 . Changed it back to StorageType[]
          Hide
          ehiggs Ewan Higgs added a comment -

          If I understand correctly, the storageType being implemented using varargs lets us omit the storageTypes parameter because it's the last parameter. However 9807 adds the StorageID which will then be the last parameter so will this approach be able to be extended?

          Show
          ehiggs Ewan Higgs added a comment - If I understand correctly, the storageType being implemented using varargs lets us omit the storageTypes parameter because it's the last parameter. However 9807 adds the StorageID which will then be the last parameter so will this approach be able to be extended?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 13m 50s trunk passed
          +1 compile 2m 16s trunk passed
          +1 checkstyle 1m 13s trunk passed
          +1 mvnsite 2m 26s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 4m 38s trunk passed
          +1 javadoc 1m 32s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 59s the patch passed
          +1 compile 2m 21s the patch passed
          +1 cc 2m 21s the patch passed
          +1 javac 2m 21s the patch passed
          -0 checkstyle 0m 54s hadoop-hdfs-project: The patch generated 1 new + 670 unchanged - 8 fixed = 671 total (was 678)
          +1 mvnsite 2m 13s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 49s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed.
          -1 unit 74m 52s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          118m 7s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863365/HDFS-6708.0009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 073be4b6c2ec 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0cab572
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19082/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19082/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19082/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19082/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 13m 50s trunk passed +1 compile 2m 16s trunk passed +1 checkstyle 1m 13s trunk passed +1 mvnsite 2m 26s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 4m 38s trunk passed +1 javadoc 1m 32s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 2m 21s the patch passed +1 cc 2m 21s the patch passed +1 javac 2m 21s the patch passed -0 checkstyle 0m 54s hadoop-hdfs-project: The patch generated 1 new + 670 unchanged - 8 fixed = 671 total (was 678) +1 mvnsite 2m 13s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 49s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 1m 10s hadoop-hdfs-client in the patch passed. -1 unit 74m 52s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 118m 7s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863365/HDFS-6708.0009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 073be4b6c2ec 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19082/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19082/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19082/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19082/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Fix checkstyle errors introduced in 0008

          Show
          chris.douglas Chris Douglas added a comment - Fix checkstyle errors introduced in 0008
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 17m 11s trunk passed
          +1 compile 18m 1s trunk passed
          +1 checkstyle 2m 13s trunk passed
          +1 mvnsite 3m 14s trunk passed
          +1 mvneclipse 1m 8s trunk passed
          +1 findbugs 5m 51s trunk passed
          +1 javadoc 2m 25s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 2m 52s the patch passed
          +1 compile 18m 0s the patch passed
          +1 cc 18m 0s the patch passed
          +1 javac 18m 0s the patch passed
          -0 checkstyle 2m 19s root: The patch generated 4 new + 673 unchanged - 8 fixed = 677 total (was 681)
          +1 mvnsite 2m 58s the patch passed
          +1 mvneclipse 1m 8s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 59s the patch passed
          +1 javadoc 2m 14s the patch passed
          -1 unit 7m 37s hadoop-common in the patch failed.
          +1 unit 1m 22s hadoop-hdfs-client in the patch passed.
          -1 unit 67m 0s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          164m 48s



          Reason Tests
          Failed junit tests hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
            hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863217/HDFS-6708.0008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 9d0ecf3bbd65 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0cab572
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19076/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19076/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19076/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19076/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19076/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 17m 11s trunk passed +1 compile 18m 1s trunk passed +1 checkstyle 2m 13s trunk passed +1 mvnsite 3m 14s trunk passed +1 mvneclipse 1m 8s trunk passed +1 findbugs 5m 51s trunk passed +1 javadoc 2m 25s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 52s the patch passed +1 compile 18m 0s the patch passed +1 cc 18m 0s the patch passed +1 javac 18m 0s the patch passed -0 checkstyle 2m 19s root: The patch generated 4 new + 673 unchanged - 8 fixed = 677 total (was 681) +1 mvnsite 2m 58s the patch passed +1 mvneclipse 1m 8s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 59s the patch passed +1 javadoc 2m 14s the patch passed -1 unit 7m 37s hadoop-common in the patch failed. +1 unit 1m 22s hadoop-hdfs-client in the patch passed. -1 unit 67m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 164m 48s Reason Tests Failed junit tests hadoop.security.token.delegation.TestZKDelegationTokenSecretManager   hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863217/HDFS-6708.0008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9d0ecf3bbd65 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19076/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19076/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19076/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19076/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19076/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          Attaching a version that uses varargs instead of Optional and omits the StorageType#asArray method. Ewan Higgs, does this look OK?

          I'll commit this (or a variant) soon.

          Show
          chris.douglas Chris Douglas added a comment - Attaching a version that uses varargs instead of Optional and omits the StorageType#asArray method. Ewan Higgs , does this look OK? I'll commit this (or a variant) soon.
          Hide
          virajith Virajith Jalaparti added a comment -

          Yes, TestDataNodeVolumeFailureReporting times out on trunk also.

          Show
          virajith Virajith Jalaparti added a comment - Yes, TestDataNodeVolumeFailureReporting times out on trunk also.
          Hide
          ehiggs Ewan Higgs added a comment -

          The only common failure between two runs of tests is TestDataNodeVolumeFailureReporting and this times out on trunk for me (in tearDown).

          Show
          ehiggs Ewan Higgs added a comment - The only common failure between two runs of tests is TestDataNodeVolumeFailureReporting and this times out on trunk for me (in tearDown ).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 33s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 39s Maven dependency ordering for branch
          +1 mvninstall 18m 7s trunk passed
          +1 compile 26m 23s trunk passed
          +1 checkstyle 2m 41s trunk passed
          +1 mvnsite 3m 54s trunk passed
          +1 mvneclipse 1m 20s trunk passed
          +1 findbugs 6m 1s trunk passed
          +1 javadoc 2m 49s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 2m 52s the patch passed
          +1 compile 23m 15s the patch passed
          +1 cc 23m 15s the patch passed
          +1 javac 23m 15s the patch passed
          -0 checkstyle 2m 44s root: The patch generated 1 new + 673 unchanged - 7 fixed = 674 total (was 680)
          +1 mvnsite 4m 0s the patch passed
          +1 mvneclipse 1m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 6m 59s the patch passed
          +1 javadoc 3m 2s the patch passed
          +1 unit 11m 22s hadoop-common in the patch passed.
          +1 unit 1m 32s hadoop-hdfs-client in the patch passed.
          -1 unit 142m 29s hadoop-hdfs in the patch failed.
          +1 asflicense 1m 5s The patch does not generate ASF License warnings.
          267m 46s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.datanode.TestDataNodeUUID



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860108/HDFS-6708.0007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 4abff6ec75ce 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 13c766b
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18896/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18896/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18896/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18896/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 39s Maven dependency ordering for branch +1 mvninstall 18m 7s trunk passed +1 compile 26m 23s trunk passed +1 checkstyle 2m 41s trunk passed +1 mvnsite 3m 54s trunk passed +1 mvneclipse 1m 20s trunk passed +1 findbugs 6m 1s trunk passed +1 javadoc 2m 49s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 2m 52s the patch passed +1 compile 23m 15s the patch passed +1 cc 23m 15s the patch passed +1 javac 23m 15s the patch passed -0 checkstyle 2m 44s root: The patch generated 1 new + 673 unchanged - 7 fixed = 674 total (was 680) +1 mvnsite 4m 0s the patch passed +1 mvneclipse 1m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 59s the patch passed +1 javadoc 3m 2s the patch passed +1 unit 11m 22s hadoop-common in the patch passed. +1 unit 1m 32s hadoop-hdfs-client in the patch passed. -1 unit 142m 29s hadoop-hdfs in the patch failed. +1 asflicense 1m 5s The patch does not generate ASF License warnings. 267m 46s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.datanode.TestDataNodeUUID Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860108/HDFS-6708.0007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 4abff6ec75ce 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13c766b Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18896/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18896/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18896/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18896/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          I triggered a rebuild, though you're right that these are probably just flaky tests:

          https://builds.apache.org/job/PreCommit-HDFS-Build/18896/

          Show
          andrew.wang Andrew Wang added a comment - I triggered a rebuild, though you're right that these are probably just flaky tests: https://builds.apache.org/job/PreCommit-HDFS-Build/18896/
          Hide
          ehiggs Ewan Higgs added a comment -

          For what it's worth, I cannot reproduce the test failures locally. I suspect there may have been some test instability. Is it possible to trigger a rebuild? I don't have permission on builds.apache.org to trigger it manually and there's no comment crawler like on Github where you can just reply "jenkins, please test this".

          Show
          ehiggs Ewan Higgs added a comment - For what it's worth, I cannot reproduce the test failures locally. I suspect there may have been some test instability. Is it possible to trigger a rebuild? I don't have permission on builds.apache.org to trigger it manually and there's no comment crawler like on Github where you can just reply "jenkins, please test this".
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 54s Maven dependency ordering for branch
          +1 mvninstall 16m 35s trunk passed
          +1 compile 23m 17s trunk passed
          +1 checkstyle 2m 12s trunk passed
          +1 mvnsite 2m 59s trunk passed
          +1 mvneclipse 1m 2s trunk passed
          +1 findbugs 6m 6s trunk passed
          +1 javadoc 2m 9s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 6s the patch passed
          +1 compile 17m 41s the patch passed
          +1 cc 17m 41s the patch passed
          +1 javac 17m 41s the patch passed
          -0 checkstyle 2m 20s root: The patch generated 1 new + 674 unchanged - 7 fixed = 675 total (was 681)
          +1 mvnsite 3m 20s the patch passed
          +1 mvneclipse 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 6m 29s the patch passed
          +1 javadoc 2m 27s the patch passed
          +1 unit 8m 27s hadoop-common in the patch passed.
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          -1 unit 81m 51s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          185m 50s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860108/HDFS-6708.0007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux eeacb8aa9d55 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 59d6925
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18811/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18811/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18811/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18811/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 54s Maven dependency ordering for branch +1 mvninstall 16m 35s trunk passed +1 compile 23m 17s trunk passed +1 checkstyle 2m 12s trunk passed +1 mvnsite 2m 59s trunk passed +1 mvneclipse 1m 2s trunk passed +1 findbugs 6m 6s trunk passed +1 javadoc 2m 9s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 6s the patch passed +1 compile 17m 41s the patch passed +1 cc 17m 41s the patch passed +1 javac 17m 41s the patch passed -0 checkstyle 2m 20s root: The patch generated 1 new + 674 unchanged - 7 fixed = 675 total (was 681) +1 mvnsite 3m 20s the patch passed +1 mvneclipse 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 29s the patch passed +1 javadoc 2m 27s the patch passed +1 unit 8m 27s hadoop-common in the patch passed. +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 81m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 185m 50s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860108/HDFS-6708.0007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux eeacb8aa9d55 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 59d6925 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18811/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18811/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18811/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18811/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching patch 0007 which fixes the Javadoc issue and minor checkstyle issue.

          Show
          ehiggs Ewan Higgs added a comment - Attaching patch 0007 which fixes the Javadoc issue and minor checkstyle issue.
          Hide
          chris.douglas Chris Douglas added a comment - - edited

          +1 This looks good to me, too (once tests pass, javadoc fix, etc.)

          Daryn Sharp do you have an opinion on backwards compatibility for the Writable case? I suspect older clients will ignore the extra fields harmlessly- these should be framed in the RPC- but I haven't actually verified.

          Show
          chris.douglas Chris Douglas added a comment - - edited +1 This looks good to me, too (once tests pass, javadoc fix, etc.) Daryn Sharp do you have an opinion on backwards compatibility for the Writable case? I suspect older clients will ignore the extra fields harmlessly- these should be framed in the RPC- but I haven't actually verified.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 12m 14s trunk passed
          +1 compile 10m 27s trunk passed
          +1 checkstyle 2m 10s trunk passed
          +1 mvnsite 2m 52s trunk passed
          +1 mvneclipse 1m 13s trunk passed
          +1 findbugs 4m 58s trunk passed
          +1 javadoc 2m 14s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 53s the patch passed
          +1 compile 10m 8s the patch passed
          +1 cc 10m 8s the patch passed
          +1 javac 10m 8s the patch passed
          -0 checkstyle 2m 8s root: The patch generated 3 new + 675 unchanged - 7 fixed = 678 total (was 682)
          +1 mvnsite 2m 49s the patch passed
          +1 mvneclipse 1m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 33s the patch passed
          -1 javadoc 0m 55s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9)
          -1 unit 8m 56s hadoop-common in the patch failed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 70m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 50s The patch does not generate ASF License warnings.
          146m 11s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.security.TestKDiag
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859166/HDFS-6708.0006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux cc08746fd8d9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 09ad8ef
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18745/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18745/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 12m 14s trunk passed +1 compile 10m 27s trunk passed +1 checkstyle 2m 10s trunk passed +1 mvnsite 2m 52s trunk passed +1 mvneclipse 1m 13s trunk passed +1 findbugs 4m 58s trunk passed +1 javadoc 2m 14s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 10m 8s the patch passed +1 cc 10m 8s the patch passed +1 javac 10m 8s the patch passed -0 checkstyle 2m 8s root: The patch generated 3 new + 675 unchanged - 7 fixed = 678 total (was 682) +1 mvnsite 2m 49s the patch passed +1 mvneclipse 1m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 33s the patch passed -1 javadoc 0m 55s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9) -1 unit 8m 56s hadoop-common in the patch failed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 70m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 50s The patch does not generate ASF License warnings. 146m 11s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.security.TestKDiag   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859166/HDFS-6708.0006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux cc08746fd8d9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 09ad8ef Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18745/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18745/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18745/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching patch with test function parameters inverted based on Virajith Jalaparti's feedback. It was indeed a silly inconsistency.

          Show
          ehiggs Ewan Higgs added a comment - Attaching patch with test function parameters inverted based on Virajith Jalaparti 's feedback. It was indeed a silly inconsistency.
          Hide
          virajith Virajith Jalaparti added a comment -

          Thanks for the quick response Ewan Higgs.

          One minor comment: could you please swap the names of the parameters (allowed, and requested) in TestBlockStoragePolicy#testStorageTypeCheckAccessResult, to be consistent with the names in BlockTokenSecretManager#checkAccess? It is a bit confusing.

          Other than that, lgtm +1

          Show
          virajith Virajith Jalaparti added a comment - Thanks for the quick response Ewan Higgs . One minor comment: could you please swap the names of the parameters ( allowed , and requested ) in TestBlockStoragePolicy#testStorageTypeCheckAccessResult , to be consistent with the names in BlockTokenSecretManager#checkAccess ? It is a bit confusing. Other than that, lgtm +1
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          -1 mvninstall 12m 59s root in trunk failed.
          +1 compile 10m 39s trunk passed
          +1 checkstyle 2m 6s trunk passed
          +1 mvnsite 2m 57s trunk passed
          +1 mvneclipse 1m 14s trunk passed
          +1 findbugs 5m 5s trunk passed
          +1 javadoc 2m 13s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 53s the patch passed
          +1 compile 10m 22s the patch passed
          +1 cc 10m 22s the patch passed
          +1 javac 10m 22s the patch passed
          -0 checkstyle 2m 8s root: The patch generated 1 new + 675 unchanged - 7 fixed = 676 total (was 682)
          +1 mvnsite 2m 50s the patch passed
          +1 mvneclipse 1m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 26s the patch passed
          -1 javadoc 0m 51s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9)
          -1 unit 8m 14s hadoop-common in the patch failed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 72m 5s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 47s The patch does not generate ASF License warnings.
          147m 56s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859114/HDFS-6708.0005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 840aabe33d29 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7114bad
          Default Java 1.8.0_121
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/branch-mvninstall-root.txt
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18738/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18738/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch -1 mvninstall 12m 59s root in trunk failed. +1 compile 10m 39s trunk passed +1 checkstyle 2m 6s trunk passed +1 mvnsite 2m 57s trunk passed +1 mvneclipse 1m 14s trunk passed +1 findbugs 5m 5s trunk passed +1 javadoc 2m 13s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 10m 22s the patch passed +1 cc 10m 22s the patch passed +1 javac 10m 22s the patch passed -0 checkstyle 2m 8s root: The patch generated 1 new + 675 unchanged - 7 fixed = 676 total (was 682) +1 mvnsite 2m 50s the patch passed +1 mvneclipse 1m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 26s the patch passed -1 javadoc 0m 51s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9) -1 unit 8m 14s hadoop-common in the patch failed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 72m 5s hadoop-hdfs in the patch failed. +1 asflicense 0m 47s The patch does not generate ASF License warnings. 147m 56s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859114/HDFS-6708.0005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 840aabe33d29 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7114bad Default Java 1.8.0_121 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18738/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18738/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18738/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching patch that reflects Virajith Jalaparti's comments.

          Show
          ehiggs Ewan Higgs added a comment - Attaching patch that reflects Virajith Jalaparti 's comments.
          Hide
          virajith Virajith Jalaparti added a comment -

          Thanks for all the work Ewan Higgs and Pieter Reuse.

          In the latest patch (v4), checkAccess(StorageType[] storageTypesRequested, StorageType[] storageTypesAvailable) in BlockTokenSecretManager passes as long as one element of storageTypesRequested is present in storageTypesAvailable. A couple of comments on this:

          1. I think the check should pass only if all elements in storageTypesAvailable are also present in storageTypesRequested. This is because the intent of this JIRA is to ensure that whatever storage types are used for block creation, the BlockTokenIdentifier should be used as a mechanism to verify them. Hence, storageTypesRequested which is determined from the BlockTokenIdentifier should be a superset of storageTypesAvailable, which are used in the block creation protocol (e.g., writeBlock).
          2. (Minor) The variable names storageTypesAvailable and storageTypesRequested are a bit confusing. storageTypesAvailable denotes the storage types that are passed in the RPC call (transferBlock, writeBlock etc.), and storageTypesRequested denotes the storage types that are specified in the BlockTokenIdentifier. It would be good to call them along those lines – say something the lines, candidateStorageTypes, and storageTypesRequested.
          Show
          virajith Virajith Jalaparti added a comment - Thanks for all the work Ewan Higgs and Pieter Reuse . In the latest patch (v4), checkAccess(StorageType[] storageTypesRequested, StorageType[] storageTypesAvailable) in BlockTokenSecretManager passes as long as one element of storageTypesRequested is present in storageTypesAvailable . A couple of comments on this: I think the check should pass only if all elements in storageTypesAvailable are also present in storageTypesRequested . This is because the intent of this JIRA is to ensure that whatever storage types are used for block creation, the BlockTokenIdentifier should be used as a mechanism to verify them. Hence, storageTypesRequested which is determined from the BlockTokenIdentifier should be a superset of storageTypesAvailable , which are used in the block creation protocol (e.g., writeBlock ). (Minor) The variable names storageTypesAvailable and storageTypesRequested are a bit confusing. storageTypesAvailable denotes the storage types that are passed in the RPC call ( transferBlock , writeBlock etc.), and storageTypesRequested denotes the storage types that are specified in the BlockTokenIdentifier . It would be good to call them along those lines – say something the lines, candidateStorageTypes , and storageTypesRequested .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 54s Maven dependency ordering for branch
          +1 mvninstall 15m 29s trunk passed
          +1 compile 16m 10s trunk passed
          +1 checkstyle 2m 12s trunk passed
          +1 mvnsite 2m 57s trunk passed
          +1 mvneclipse 1m 2s trunk passed
          +1 findbugs 5m 26s trunk passed
          +1 javadoc 2m 8s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 12s the patch passed
          +1 compile 13m 21s the patch passed
          +1 cc 13m 21s the patch passed
          +1 javac 13m 21s the patch passed
          -0 checkstyle 2m 18s root: The patch generated 1 new + 674 unchanged - 7 fixed = 675 total (was 681)
          +1 mvnsite 3m 23s the patch passed
          +1 mvneclipse 1m 1s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 6m 36s the patch passed
          -1 javadoc 0m 56s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 7 unchanged - 0 fixed = 10 total (was 7)
          +1 unit 9m 41s hadoop-common in the patch passed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 74m 16s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          166m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855364/HDFS-6708.0004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux fe2574e7f0a8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 82ef9ac
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18487/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18487/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18487/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18487/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18487/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 54s Maven dependency ordering for branch +1 mvninstall 15m 29s trunk passed +1 compile 16m 10s trunk passed +1 checkstyle 2m 12s trunk passed +1 mvnsite 2m 57s trunk passed +1 mvneclipse 1m 2s trunk passed +1 findbugs 5m 26s trunk passed +1 javadoc 2m 8s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 12s the patch passed +1 compile 13m 21s the patch passed +1 cc 13m 21s the patch passed +1 javac 13m 21s the patch passed -0 checkstyle 2m 18s root: The patch generated 1 new + 674 unchanged - 7 fixed = 675 total (was 681) +1 mvnsite 3m 23s the patch passed +1 mvneclipse 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 36s the patch passed -1 javadoc 0m 56s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 7 unchanged - 0 fixed = 10 total (was 7) +1 unit 9m 41s hadoop-common in the patch passed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 74m 16s hadoop-hdfs in the patch failed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 166m 28s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855364/HDFS-6708.0004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux fe2574e7f0a8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 82ef9ac Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18487/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18487/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18487/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18487/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18487/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching a patch that allows the token check to pass if any of the StorageTypes on the DN are available. This is because a Datanode with only SSD only needs to satisfy the SSD part of the request; not SSD and DISK since the other DNs need to handle the DISK replicas.

          This will be more obviously required when we check StorageIDs in HDFS-9807.

          Show
          ehiggs Ewan Higgs added a comment - Attaching a patch that allows the token check to pass if any of the StorageTypes on the DN are available. This is because a Datanode with only SSD only needs to satisfy the SSD part of the request; not SSD and DISK since the other DNs need to handle the DISK replicas. This will be more obviously required when we check StorageIDs in HDFS-9807 .
          Hide
          ehiggs Ewan Higgs added a comment -

          Hi,
          Could I get a review on this if/when someone has time? This is slowing HDFS-9807 since they touch the same code.

          Based on the people who previously reviewed changes to the BlockTokenIdenfier, I think Chris Douglas, Andrew Wang, and Daryn Sharp are good candidates for review. Of course, Arpit Agarwal as well as he's the reporter.

          Thanks

          Show
          ehiggs Ewan Higgs added a comment - Hi, Could I get a review on this if/when someone has time? This is slowing HDFS-9807 since they touch the same code. Based on the people who previously reviewed changes to the BlockTokenIdenfier, I think Chris Douglas , Andrew Wang , and Daryn Sharp are good candidates for review. Of course, Arpit Agarwal as well as he's the reporter. Thanks
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 38s Maven dependency ordering for branch
          +1 mvninstall 15m 46s trunk passed
          +1 compile 13m 23s trunk passed
          +1 checkstyle 1m 44s trunk passed
          +1 mvnsite 2m 37s trunk passed
          +1 mvneclipse 1m 0s trunk passed
          +1 findbugs 4m 54s trunk passed
          +1 javadoc 1m 58s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 52s the patch passed
          +1 compile 11m 5s the patch passed
          +1 cc 11m 5s the patch passed
          +1 javac 11m 5s the patch passed
          -0 checkstyle 1m 41s root: The patch generated 1 new + 690 unchanged - 7 fixed = 691 total (was 697)
          +1 mvnsite 2m 32s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 8s the patch passed
          -1 javadoc 0m 44s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 7 unchanged - 0 fixed = 10 total (was 7)
          +1 unit 8m 36s hadoop-common in the patch passed.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed.
          -1 unit 67m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          148m 25s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852593/HDFS-6708.0003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 7932de80a8f1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aaf106f
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18372/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18372/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18372/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18372/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18372/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 38s Maven dependency ordering for branch +1 mvninstall 15m 46s trunk passed +1 compile 13m 23s trunk passed +1 checkstyle 1m 44s trunk passed +1 mvnsite 2m 37s trunk passed +1 mvneclipse 1m 0s trunk passed +1 findbugs 4m 54s trunk passed +1 javadoc 1m 58s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 11m 5s the patch passed +1 cc 11m 5s the patch passed +1 javac 11m 5s the patch passed -0 checkstyle 1m 41s root: The patch generated 1 new + 690 unchanged - 7 fixed = 691 total (was 697) +1 mvnsite 2m 32s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 8s the patch passed -1 javadoc 0m 44s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 7 unchanged - 0 fixed = 10 total (was 7) +1 unit 8m 36s hadoop-common in the patch passed. +1 unit 1m 1s hadoop-hdfs-client in the patch passed. -1 unit 67m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 148m 25s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852593/HDFS-6708.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 7932de80a8f1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aaf106f Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18372/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18372/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18372/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18372/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18372/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching a rebase of the patch on top of HDFS-11026 (which has been merged).

          Show
          ehiggs Ewan Higgs added a comment - Attaching a rebase of the patch on top of HDFS-11026 (which has been merged).
          Hide
          ehiggs Ewan Higgs added a comment -

          Attached is a second version of the patch which correctly computes the hash of the storageTypes array.

          Show
          ehiggs Ewan Higgs added a comment - Attached is a second version of the patch which correctly computes the hash of the storageTypes array.
          Hide
          ehiggs Ewan Higgs added a comment -

          The javadoc errors are due to a longstanding bug where javadoc seems to be unable to process generic arguments in link or see tags: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6435888

          The findbugs error was a good catch. I used storageTypes.hashCode() but need Arrays.hashCode(storageTypes)

          Show
          ehiggs Ewan Higgs added a comment - The javadoc errors are due to a longstanding bug where javadoc seems to be unable to process generic arguments in link or see tags: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6435888 The findbugs error was a good catch. I used storageTypes.hashCode() but need Arrays.hashCode(storageTypes)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 50s Maven dependency ordering for branch
          +1 mvninstall 15m 58s trunk passed
          +1 compile 15m 27s trunk passed
          +1 checkstyle 1m 48s trunk passed
          +1 mvnsite 2m 58s trunk passed
          +1 mvneclipse 1m 4s trunk passed
          +1 findbugs 5m 35s trunk passed
          +1 javadoc 2m 17s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 2m 50s the patch passed
          +1 compile 14m 58s the patch passed
          +1 javac 14m 58s the patch passed
          -0 checkstyle 1m 59s root: The patch generated 1 new + 648 unchanged - 7 fixed = 649 total (was 655)
          +1 mvnsite 3m 26s the patch passed
          +1 mvneclipse 0m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 2m 4s hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 48s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 7 unchanged - 0 fixed = 10 total (was 7)
          +1 unit 8m 35s hadoop-common in the patch passed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          +1 unit 72m 55s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 45s The patch does not generate ASF License warnings.
          164m 13s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client
            Invocation of hashCode on array in org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.hashCode() At BlockTokenIdentifier.java:in org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.hashCode() At BlockTokenIdentifier.java:[line 156]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-6708
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850395/HDFS-6708.0001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8d56f6161db4 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0914fcc
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18323/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18323/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18323/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18323/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18323/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 50s Maven dependency ordering for branch +1 mvninstall 15m 58s trunk passed +1 compile 15m 27s trunk passed +1 checkstyle 1m 48s trunk passed +1 mvnsite 2m 58s trunk passed +1 mvneclipse 1m 4s trunk passed +1 findbugs 5m 35s trunk passed +1 javadoc 2m 17s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 50s the patch passed +1 compile 14m 58s the patch passed +1 javac 14m 58s the patch passed -0 checkstyle 1m 59s root: The patch generated 1 new + 648 unchanged - 7 fixed = 649 total (was 655) +1 mvnsite 3m 26s the patch passed +1 mvneclipse 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 2m 4s hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 48s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 7 unchanged - 0 fixed = 10 total (was 7) +1 unit 8m 35s hadoop-common in the patch passed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. +1 unit 72m 55s hadoop-hdfs in the patch passed. +1 asflicense 0m 45s The patch does not generate ASF License warnings. 164m 13s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client   Invocation of hashCode on array in org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.hashCode() At BlockTokenIdentifier.java:in org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.hashCode() At BlockTokenIdentifier.java: [line 156] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-6708 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850395/HDFS-6708.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8d56f6161db4 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0914fcc Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18323/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18323/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18323/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18323/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18323/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          If HDFS-11026 is merged, this will need a rebase which I am happy to do.

          Show
          ehiggs Ewan Higgs added a comment - If HDFS-11026 is merged, this will need a rebase which I am happy to do.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attached is a patch that will check the StorageType of the writes as described in the previous post.

          A lot of this work was done by Pieter Reuse.

          Show
          ehiggs Ewan Higgs added a comment - Attached is a patch that will check the StorageType of the writes as described in the previous post. A lot of this work was done by Pieter Reuse .
          Hide
          ehiggs Ewan Higgs added a comment - - edited

          Hi all. I took this work over from Pieter. I've got some questions about how we want this to work.

          First, if I understand correctly we want to check the token mainly in the DataXceiver, passing the token and the locally available StorageTypes whenever we call DataXceiver.checkAccess (which ends up calling BlockTokenSecretManager.checkAccess) for writes only (i.e. replaceBlock, transferBlock, writeBlock - and not blockChecksum, blockGroupChecksum, or requestShortCircuitFds).

          If we're only checking the storage type when we write data then we use the passed in StorageType. However, if we want to do this on reads as well, we need to gather the available StorageType[]. We can get the locally available storage types as follows:

            // only needed if we want to use StorageType for reads as well...
            private static StorageType[] getStorageTypes(DataNode datanode) {
              final FsDatasetSpi dataset =  datanode.getFSDataset();
              final FsDatasetSpi.FsVolumeReferences vols;
              vols = dataset.getFsVolumeReferences();
          
              List<StorageType> storageTypes = new ArrayList<StorageType>(vols.size());
              Iterator<FsVolumeSpi> iter = vols.iterator();
              while (iter.hasNext()) {
                FsVolumeSpi vol = iter.next();
                StorageType storageType = vol.getStorageType();
                if (storageType != null) {
                  storageTypes.add(storageType);
                }
              }
              return storageTypes.toArray(new StorageType[0]);
            }
          

          The resulting check uses the Token StorageType[] and compares it to the StorageType[] passed in by the Protobuf request operation. I think the rules should be as follows:

          Token StorageType[] Node StorageType[] Result
          * null Error
          null * Error
          [] [] Not OK (maybe Error?)
          [DISK] [DISK] OK
          [DISK] [] Not OK
          [SSD, DISK] [DISK] Not OK
          [SSD, DISK] [SSD, DISK] OK
          [] [SSD, DISK] OK
          Show
          ehiggs Ewan Higgs added a comment - - edited Hi all. I took this work over from Pieter. I've got some questions about how we want this to work. First, if I understand correctly we want to check the token mainly in the DataXceiver, passing the token and the locally available StorageTypes whenever we call DataXceiver.checkAccess (which ends up calling BlockTokenSecretManager.checkAccess ) for writes only (i.e. replaceBlock , transferBlock , writeBlock - and not blockChecksum , blockGroupChecksum , or requestShortCircuitFds ). If we're only checking the storage type when we write data then we use the passed in StorageType . However, if we want to do this on reads as well, we need to gather the available StorageType[] . We can get the locally available storage types as follows: // only needed if we want to use StorageType for reads as well... private static StorageType[] getStorageTypes(DataNode datanode) { final FsDatasetSpi dataset = datanode.getFSDataset(); final FsDatasetSpi.FsVolumeReferences vols; vols = dataset.getFsVolumeReferences(); List<StorageType> storageTypes = new ArrayList<StorageType>(vols.size()); Iterator<FsVolumeSpi> iter = vols.iterator(); while (iter.hasNext()) { FsVolumeSpi vol = iter.next(); StorageType storageType = vol.getStorageType(); if (storageType != null ) { storageTypes.add(storageType); } } return storageTypes.toArray( new StorageType[0]); } The resulting check uses the Token StorageType[] and compares it to the StorageType[] passed in by the Protobuf request operation. I think the rules should be as follows: Token StorageType[] Node StorageType[] Result * null Error null * Error [] [] Not OK (maybe Error?) [DISK] [DISK] OK [DISK] [] Not OK [SSD, DISK] [DISK] Not OK [SSD, DISK] [SSD, DISK] OK [] [SSD, DISK] OK
          Hide
          PieterReuse Pieter Reuse added a comment -

          The changes in the protobuf for the BlockTokenIdentifier will be discussed and potentially implemented in HDFS-11026. I've made great progress in implementing a patch for this ticket, but will rebase implementation on top of the patch for HDFS-11026 before uploading it here (avoiding double review-work).

          Show
          PieterReuse Pieter Reuse added a comment - The changes in the protobuf for the BlockTokenIdentifier will be discussed and potentially implemented in HDFS-11026 . I've made great progress in implementing a patch for this ticket, but will rebase implementation on top of the patch for HDFS-11026 before uploading it here (avoiding double review-work).
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I added you as a project contributor and assigned it to you. Proper typing will be an incompatible change and probably too disruptive to be practical. IIUC there's many dependencies across HDFS/Yarn that would need to be updated.

          Show
          arpitagarwal Arpit Agarwal added a comment - I added you as a project contributor and assigned it to you. Proper typing will be an incompatible change and probably too disruptive to be practical. IIUC there's many dependencies across HDFS/Yarn that would need to be updated.
          Hide
          PieterReuse Pieter Reuse added a comment -

          I'm working on a patch for this ticket, but Jira doesn't let me assign this issue to me.

          While implementing this patch, I noticed that in hadoop-common-project/hadoop-common/src/main/proto/Security.proto, the TokenProto contains a byte array for the BlockTokenIdentifier instead of a proper typed protobuf-message. Would it make sense to introduce this proper typing in this patch or does anyone disagree with this?

          Show
          PieterReuse Pieter Reuse added a comment - I'm working on a patch for this ticket, but Jira doesn't let me assign this issue to me. While implementing this patch, I noticed that in hadoop-common-project/hadoop-common/src/main/proto/Security.proto, the TokenProto contains a byte array for the BlockTokenIdentifier instead of a proper typed protobuf-message. Would it make sense to introduce this proper typing in this patch or does anyone disagree with this?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Pieter Reuse, no I haven't. I've unassigned this in case you or someone else wants to take a crack at it.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Pieter Reuse , no I haven't. I've unassigned this in case you or someone else wants to take a crack at it.
          Hide
          PieterReuse Pieter Reuse added a comment -

          Hi Arpit Agarwal, we found this ticket while looking at HDFS-9806. It looks like this ticket should be implemented for block replicas on external storages. Did you have a chance to look at this?

          Show
          PieterReuse Pieter Reuse added a comment - Hi Arpit Agarwal , we found this ticket while looking at HDFS-9806 . It looks like this ticket should be implemented for block replicas on external storages. Did you have a chance to look at this?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Nicholas, I think so. Since the storage types are not encoded in the block token the client could override the quota checks made by NN. Let me take a look at it for 2.8.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Nicholas, I think so. Since the storage types are not encoded in the block token the client could override the quota checks made by NN. Let me take a look at it for 2.8.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Hi Arpit, do we still need this JIRA?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Hi Arpit, do we still need this JIRA?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Nicholas, I think it is fine to have this as a subtask of HDFS-5682 since quota mechanism will be implemented as part of that Jira.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Nicholas, I think it is fine to have this as a subtask of HDFS-5682 since quota mechanism will be implemented as part of that Jira.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          We indeed already have a subtask HDFS-5391 for changing writeBlock and block token, i.e. HDFS-6702 + HDFS-6708 (this JIRA). Let me resolve HDFS-5391 as duplicated and move this as a subtask of HDFS-5682.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - We indeed already have a subtask HDFS-5391 for changing writeBlock and block token, i.e. HDFS-6702 + HDFS-6708 (this JIRA). Let me resolve HDFS-5391 as duplicated and move this as a subtask of HDFS-5682 .
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          We should move this as a subtask of HDFS-5682 since it is going to change quota to per storage type. Sound good?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - We should move this as a subtask of HDFS-5682 since it is going to change quota to per storage type. Sound good?

            People

            • Assignee:
              ehiggs Ewan Higgs
              Reporter:
              arpitagarwal Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development