Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The BlockPlacementPolicy considers specific storages, but when the replica is written the DN VolumeChoosingPolicy is unaware of any preference or constraints from other policies affecting placement. This limits heterogeneity to the declared storage types, which are treated as fungible within the target DN. It should be possible to influence or constrain the DN policy to select a particular storage.

      1. HDFS-9807.001.patch
        139 kB
        Ewan Higgs
      2. HDFS-9807.002.patch
        131 kB
        Chris Douglas
      3. HDFS-9807.003.patch
        147 kB
        Ewan Higgs
      4. HDFS-9807.004.patch
        147 kB
        Ewan Higgs
      5. HDFS-9807.005.patch
        148 kB
        Ewan Higgs
      6. HDFS-9807.006.patch
        128 kB
        Ewan Higgs
      7. HDFS-9807.007.patch
        143 kB
        Ewan Higgs
      8. HDFS-9807.008.patch
        143 kB
        Virajith Jalaparti
      9. HDFS-9807.009.patch
        143 kB
        Virajith Jalaparti
      10. HDFS-9807.010.patch
        141 kB
        Virajith Jalaparti

        Issue Links

          Activity

          Hide
          ehiggs Ewan Higgs added a comment -

          Should the StorageID be in the write messages (OpWriteBlockProto OpTransferBlockProto, OpReplaceBlockProto, OpCopyBlockProto) as well as the BlockTokenIdentifier, or just the messages?

          Show
          ehiggs Ewan Higgs added a comment - Should the StorageID be in the write messages ( OpWriteBlockProto OpTransferBlockProto , OpReplaceBlockProto , OpCopyBlockProto ) as well as the BlockTokenIdentifier , or just the messages?
          Hide
          ehiggs Ewan Higgs added a comment -

          To answer the previous question: I'm adding the storage ID to all the messages that also send a targetStorageType (i.e. OpWriteBlockProto,OpTransferBlockProto, OpReplaceBlockProto, and not OpCopyBlockProto).

          The storageID should also go into the BlockTokenIdentifier. With HDFS-11026 merged, I think it's best to only put this into the protobuf version. But I am open to listening to arguments for putting it in the Legacy token.

          Show
          ehiggs Ewan Higgs added a comment - To answer the previous question: I'm adding the storage ID to all the messages that also send a targetStorageType (i.e. OpWriteBlockProto , OpTransferBlockProto , OpReplaceBlockProto , and not OpCopyBlockProto ). The storageID should also go into the BlockTokenIdentifier . With HDFS-11026 merged, I think it's best to only put this into the protobuf version. But I am open to listening to arguments for putting it in the Legacy token.
          Hide
          ehiggs Ewan Higgs added a comment -

          The most obvious way to implement this is for the NN to choose a volume and have the DN do what it's told. But the NN doesn't choose a precise volume. It only uses the first Storage of a given StorageType (see DataNodeDescriptor.chooseStorage4Block). This means that all blocks would only be written to the first volume instead of spread around.

          We can either change the algorithm in DataNodeDescriptor to choose a better selection. Or, Chris Douglas, did you have a different policy in mind when you wrote the ticket? Perhaps passing the Storage ID into FsVolumeList.getNextVolume and ignoring it if the StorageType is not PROVIDED?

          Show
          ehiggs Ewan Higgs added a comment - The most obvious way to implement this is for the NN to choose a volume and have the DN do what it's told. But the NN doesn't choose a precise volume. It only uses the first Storage of a given StorageType (see DataNodeDescriptor.chooseStorage4Block ). This means that all blocks would only be written to the first volume instead of spread around. We can either change the algorithm in DataNodeDescriptor to choose a better selection. Or, Chris Douglas , did you have a different policy in mind when you wrote the ticket? Perhaps passing the Storage ID into FsVolumeList.getNextVolume and ignoring it if the StorageType is not PROVIDED ?
          Hide
          ehiggs Ewan Higgs added a comment -

          Relevant tickets:

          HDFS-8946 - current logic of chooseStorage4Block.

          HDFS-11419 - efficiency concerns of Storage selection.

          Show
          ehiggs Ewan Higgs added a comment - Relevant tickets: HDFS-8946 - current logic of chooseStorage4Block . HDFS-11419 - efficiency concerns of Storage selection.
          Hide
          jpallas Joe Pallas added a comment -

          How do you envision this interacting with VolumeChoosingPolicy? Would it go away completely, since the policy then effectively moves to the NN? Should a DN be able to override the client's choice of volume?

          How would this fit in with federation? With multiple block pools, a given NN has incomplete information about the storage being managed by a datanode. Would the name node be able to make good decisions with the information that it has available?

          Show
          jpallas Joe Pallas added a comment - How do you envision this interacting with VolumeChoosingPolicy ? Would it go away completely, since the policy then effectively moves to the NN? Should a DN be able to override the client's choice of volume? How would this fit in with federation? With multiple block pools, a given NN has incomplete information about the storage being managed by a datanode. Would the name node be able to make good decisions with the information that it has available?
          Hide
          ehiggs Ewan Higgs added a comment -

          How do you envision this interacting with VolumeChoosingPolicy?

          Joe Pallas, I am planning to push the information like storageID and storageType FsVolumeList (as described in the last suggestion in my previous comment) and probably also into the interface for the VolumeChoosingPolicy, so it can decide to choose the volume based on all the info. Currently, storageID will be thrown away since the NN doesn't send anything useful. But with PROVIDED storage (HDFS-9806), it's important to use the specified storageID.

          How would this fit in with federation? With multiple block pools, a given NN has incomplete information about the storage being managed by a datanode. Would the name node be able to make good decisions with the information that it has available?

          Given the above, I don't plan to change the NN's DataNodeDescriptor.chooseStorage4Block in this change set.

          Show
          ehiggs Ewan Higgs added a comment - How do you envision this interacting with VolumeChoosingPolicy? Joe Pallas , I am planning to push the information like storageID and storageType FsVolumeList (as described in the last suggestion in my previous comment) and probably also into the interface for the VolumeChoosingPolicy , so it can decide to choose the volume based on all the info. Currently, storageID will be thrown away since the NN doesn't send anything useful. But with PROVIDED storage ( HDFS-9806 ), it's important to use the specified storageID. How would this fit in with federation? With multiple block pools, a given NN has incomplete information about the storage being managed by a datanode. Would the name node be able to make good decisions with the information that it has available? Given the above, I don't plan to change the NN's DataNodeDescriptor.chooseStorage4Block in this change set.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching a patch that threads the StorageID all the way through to the FsVolumeList and then ignores it (since the NN isn't sending a useful value).

          Show
          ehiggs Ewan Higgs added a comment - Attaching a patch that threads the StorageID all the way through to the FsVolumeList and then ignores it (since the NN isn't sending a useful value).
          Hide
          ehiggs Ewan Higgs added a comment -

          Attached a patch that threads the StorageID all the way through to the FsVolumeList and then ignores it (since the NN isn't sending a useful value).

          Show
          ehiggs Ewan Higgs added a comment - Attached a patch that threads the StorageID all the way through to the FsVolumeList and then ignores it (since the NN isn't sending a useful value).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 1m 55s Maven dependency ordering for branch
          +1 mvninstall 12m 50s trunk passed
          +1 compile 13m 24s trunk passed
          +1 checkstyle 2m 17s trunk passed
          +1 mvnsite 2m 36s trunk passed
          +1 mvneclipse 0m 55s trunk passed
          +1 findbugs 4m 43s trunk passed
          +1 javadoc 1m 58s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 53s the patch passed
          +1 compile 11m 55s the patch passed
          +1 cc 11m 55s the patch passed
          +1 javac 11m 55s the patch passed
          -0 checkstyle 2m 15s root: The patch generated 40 new + 1766 unchanged - 27 fixed = 1806 total (was 1793)
          +1 mvnsite 2m 34s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          -1 javadoc 0m 46s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 7 unchanged - 0 fixed = 12 total (was 7)
          -1 unit 8m 17s hadoop-common in the patch failed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 71m 2s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          150m 5s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(ExtendedBlock, StorageType, Token, String, DatanodeInfo, String) may fail to close stream At DataXceiver.java:String) may fail to close stream At DataXceiver.java:[line 1165]
            Dead store to storageType in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.getNextTransientVolume(long) At FsVolumeList.java:org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.getNextTransientVolume(long) At FsVolumeList.java:[line 135]
          Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855409/HDFS-9807.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 7ae8cb6b3587 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 82ef9ac
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18491/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18491/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 1m 55s Maven dependency ordering for branch +1 mvninstall 12m 50s trunk passed +1 compile 13m 24s trunk passed +1 checkstyle 2m 17s trunk passed +1 mvnsite 2m 36s trunk passed +1 mvneclipse 0m 55s trunk passed +1 findbugs 4m 43s trunk passed +1 javadoc 1m 58s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 11m 55s the patch passed +1 cc 11m 55s the patch passed +1 javac 11m 55s the patch passed -0 checkstyle 2m 15s root: The patch generated 40 new + 1766 unchanged - 27 fixed = 1806 total (was 1793) +1 mvnsite 2m 34s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) -1 javadoc 0m 46s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 7 unchanged - 0 fixed = 12 total (was 7) -1 unit 8m 17s hadoop-common in the patch failed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 71m 2s hadoop-hdfs in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 150m 5s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(ExtendedBlock, StorageType, Token, String, DatanodeInfo, String) may fail to close stream At DataXceiver.java:String) may fail to close stream At DataXceiver.java: [line 1165]   Dead store to storageType in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.getNextTransientVolume(long) At FsVolumeList.java:org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.getNextTransientVolume(long) At FsVolumeList.java: [line 135] Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855409/HDFS-9807.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 7ae8cb6b3587 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 82ef9ac Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18491/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18491/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18491/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          chris.douglas Chris Douglas added a comment -

          I am planning to push the information like storageID and storageType FsVolumeList (as described in the last suggestion in my previous comment) and probably also into the interface for the VolumeChoosingPolicy, so it can decide to choose the volume based on all the info.

          +1 for this approach. The configured VolumeChoosingPolicy can elect to ignore the particular storageid chosen by the NameNode, as it does now.

          Uploaded a new patch with really minor changes (removing some vestigial code, fixed findbugs, etc.). A couple questions:

          • The BlockTokenIdentifier also updates the legacy, Writable token. This wouldn't work with older clients?
          • There's an assertion in StripedWriter that targetStorageIDs is not null; does that always hold?
          Show
          chris.douglas Chris Douglas added a comment - I am planning to push the information like storageID and storageType FsVolumeList (as described in the last suggestion in my previous comment) and probably also into the interface for the VolumeChoosingPolicy, so it can decide to choose the volume based on all the info. +1 for this approach. The configured VolumeChoosingPolicy can elect to ignore the particular storageid chosen by the NameNode, as it does now. Uploaded a new patch with really minor changes (removing some vestigial code, fixed findbugs, etc.). A couple questions: The BlockTokenIdentifier also updates the legacy, Writable token. This wouldn't work with older clients? There's an assertion in StripedWriter that targetStorageIDs is not null; does that always hold?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 32s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 18 new or modified test files.
          0 mvndep 2m 20s Maven dependency ordering for branch
          +1 mvninstall 13m 39s trunk passed
          +1 compile 10m 55s trunk passed
          +1 checkstyle 2m 21s trunk passed
          +1 mvnsite 3m 0s trunk passed
          +1 mvneclipse 1m 9s trunk passed
          +1 findbugs 5m 6s trunk passed
          +1 javadoc 2m 18s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 57s the patch passed
          +1 compile 10m 14s the patch passed
          +1 cc 10m 14s the patch passed
          +1 javac 10m 14s the patch passed
          -0 checkstyle 2m 19s root: The patch generated 36 new + 1756 unchanged - 26 fixed = 1792 total (was 1782)
          +1 mvnsite 2m 54s the patch passed
          +1 mvneclipse 1m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 25s the patch passed
          -1 javadoc 0m 52s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9)
          +1 unit 8m 3s hadoop-common in the patch passed.
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          -1 unit 89m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 46s The patch does not generate ASF License warnings.
          169m 19s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859208/HDFS-9807.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 7cd46f88f1f0 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c04fb35
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18747/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18747/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18747/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18747/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18747/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 18 new or modified test files. 0 mvndep 2m 20s Maven dependency ordering for branch +1 mvninstall 13m 39s trunk passed +1 compile 10m 55s trunk passed +1 checkstyle 2m 21s trunk passed +1 mvnsite 3m 0s trunk passed +1 mvneclipse 1m 9s trunk passed +1 findbugs 5m 6s trunk passed +1 javadoc 2m 18s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 57s the patch passed +1 compile 10m 14s the patch passed +1 cc 10m 14s the patch passed +1 javac 10m 14s the patch passed -0 checkstyle 2m 19s root: The patch generated 36 new + 1756 unchanged - 26 fixed = 1792 total (was 1782) +1 mvnsite 2m 54s the patch passed +1 mvneclipse 1m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 25s the patch passed -1 javadoc 0m 52s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9) +1 unit 8m 3s hadoop-common in the patch passed. +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 89m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 46s The patch does not generate ASF License warnings. 169m 19s Reason Tests Failed junit tests hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859208/HDFS-9807.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 7cd46f88f1f0 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c04fb35 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18747/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18747/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18747/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18747/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18747/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          virajith Virajith Jalaparti added a comment -

          Thanks Ewan Higgs, and Chris Douglas for posting the patches. A couple of comments:
          a) The latest patch does not pass in the storageID to the VolumeChoosingPolicy; it is ignored after it is passed into FsVolumeList#getNextVolume. I think the interface of VolumeChoosingPolicy#chooseVolume should be modified to have storageID as one of the parameters. Then, it will be able to decide if it respects the chosen storageID or not.

          b) I think the BlockTokenSecretManager#checkAccess(String[] storageIdsRequested, String[] storageIdsAvailable) needs to be changed in a manner similar to this comment on HDFS-6708 i.e., the check should pass if and only if the storage ids in storageIdsAvailable are a subset of those in storageIdsRequested.

          Also, I assume once HDFS-6708 goes into trunk, this patch would be updated to only contain changes required for adding storage ids in the write protocol and block token identifier, and not storage types.

          Show
          virajith Virajith Jalaparti added a comment - Thanks Ewan Higgs , and Chris Douglas for posting the patches. A couple of comments: a) The latest patch does not pass in the storageID to the VolumeChoosingPolicy ; it is ignored after it is passed into FsVolumeList#getNextVolume . I think the interface of VolumeChoosingPolicy#chooseVolume should be modified to have storageID as one of the parameters. Then, it will be able to decide if it respects the chosen storageID or not. b) I think the BlockTokenSecretManager#checkAccess(String[] storageIdsRequested, String[] storageIdsAvailable) needs to be changed in a manner similar to this comment on HDFS-6708 i.e., the check should pass if and only if the storage ids in storageIdsAvailable are a subset of those in storageIdsRequested . Also, I assume once HDFS-6708 goes into trunk, this patch would be updated to only contain changes required for adding storage ids in the write protocol and block token identifier, and not storage types.
          Hide
          ehiggs Ewan Higgs added a comment -

          +1 on all of Virajith Jalaparti's comments. I think the comment about VolumeChoosingPolicy is what Chris Douglas had said and I can make that change.

          Also, I assume once HDFS-6708 goes into trunk, this patch would be updated to only contain changes required for adding storage ids in the write protocol and block token identifier, and not storage types.

          Yes, I have internal branches and can recut a patch if/shen HDFS-6708 is merged.

          Show
          ehiggs Ewan Higgs added a comment - +1 on all of Virajith Jalaparti 's comments. I think the comment about VolumeChoosingPolicy is what Chris Douglas had said and I can make that change. Also, I assume once HDFS-6708 goes into trunk, this patch would be updated to only contain changes required for adding storage ids in the write protocol and block token identifier, and not storage types. Yes, I have internal branches and can recut a patch if/shen HDFS-6708 is merged.
          Hide
          ehiggs Ewan Higgs added a comment -

          The BlockTokenIdentifier also updates the legacy, Writable token. This wouldn't work with older clients?

          I believe it will work but have not written a test that bridges the versions. If cross version testing is a requirement to getting this in, I would prefer to only support the protobuf version since that was one of the major benefits of the protobufification of the BlockTokenIdentifier.

          There are a few scenarios:

          1. Old client doesn't know about StorageTypes or StorageIDs in the BTI. I think this should work because the older BTI will read the fields it is interested in and then stop. I haven't seen any code that will complain if the entire DataInputStream isn't consumed.

          2. If the old client knows about StorageTypes but not StorageIDs then it should work (i.e. HDFS-6708 is merged but HDFS-9807 is not). The old client will read the vint for StorageTypes and then the StorageTypes and then stop.

          There's an assertion in StripedWriter that targetStorageIDs is not null; does that always hold?

          I believe it should always hold. If, e.g., the DN is upgraded before the NN and no targetStorageIDs are sent then the message will have no targetStorageIDs. This is represented as an empty list rather than null.

          Show
          ehiggs Ewan Higgs added a comment - The BlockTokenIdentifier also updates the legacy, Writable token. This wouldn't work with older clients? I believe it will work but have not written a test that bridges the versions. If cross version testing is a requirement to getting this in, I would prefer to only support the protobuf version since that was one of the major benefits of the protobufification of the BlockTokenIdentifier . There are a few scenarios: 1. Old client doesn't know about StorageTypes or StorageIDs in the BTI. I think this should work because the older BTI will read the fields it is interested in and then stop. I haven't seen any code that will complain if the entire DataInputStream isn't consumed. 2. If the old client knows about StorageTypes but not StorageIDs then it should work (i.e. HDFS-6708 is merged but HDFS-9807 is not). The old client will read the vint for StorageTypes and then the StorageTypes and then stop. There's an assertion in StripedWriter that targetStorageIDs is not null; does that always hold? I believe it should always hold. If, e.g., the DN is upgraded before the NN and no targetStorageIDs are sent then the message will have no targetStorageIDs . This is represented as an empty list rather than null.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching file that rebases on top of the updated HDFS-6708.006.patch and takes Virajith Jalaparti and Chris Douglas's comments into account.

          Show
          ehiggs Ewan Higgs added a comment - Attaching file that rebases on top of the updated HDFS-6708 .006.patch and takes Virajith Jalaparti and Chris Douglas 's comments into account.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 1m 55s Maven dependency ordering for branch
          +1 mvninstall 16m 25s trunk passed
          +1 compile 26m 34s trunk passed
          +1 checkstyle 2m 48s trunk passed
          +1 mvnsite 3m 53s trunk passed
          +1 mvneclipse 1m 8s trunk passed
          +1 findbugs 6m 45s trunk passed
          +1 javadoc 2m 40s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 59s the patch passed
          +1 compile 17m 47s the patch passed
          +1 cc 17m 47s the patch passed
          +1 javac 17m 47s the patch passed
          -0 checkstyle 2m 28s root: The patch generated 39 new + 1763 unchanged - 29 fixed = 1802 total (was 1792)
          +1 mvnsite 2m 51s the patch passed
          +1 mvneclipse 1m 3s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 2m 14s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 50s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9)
          -1 unit 7m 55s hadoop-common in the patch failed.
          +1 unit 1m 14s hadoop-hdfs-client in the patch passed.
          -1 unit 79m 1s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          188m 25s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Dead store to ident in org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(ExtendedBlock, StorageType, Token, String, DatanodeInfo, String) At DataXceiver.java:org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(ExtendedBlock, StorageType, Token, String, DatanodeInfo, String) At DataXceiver.java:[line 1163]
          Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859927/HDFS-9807.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux d76d37387cee 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f462e1f
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18798/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18798/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 1m 55s Maven dependency ordering for branch +1 mvninstall 16m 25s trunk passed +1 compile 26m 34s trunk passed +1 checkstyle 2m 48s trunk passed +1 mvnsite 3m 53s trunk passed +1 mvneclipse 1m 8s trunk passed +1 findbugs 6m 45s trunk passed +1 javadoc 2m 40s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 59s the patch passed +1 compile 17m 47s the patch passed +1 cc 17m 47s the patch passed +1 javac 17m 47s the patch passed -0 checkstyle 2m 28s root: The patch generated 39 new + 1763 unchanged - 29 fixed = 1802 total (was 1792) +1 mvnsite 2m 51s the patch passed +1 mvneclipse 1m 3s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 2m 14s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 50s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9) -1 unit 7m 55s hadoop-common in the patch failed. +1 unit 1m 14s hadoop-hdfs-client in the patch passed. -1 unit 79m 1s hadoop-hdfs in the patch failed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 188m 25s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Dead store to ident in org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(ExtendedBlock, StorageType, Token, String, DatanodeInfo, String) At DataXceiver.java:org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(ExtendedBlock, StorageType, Token, String, DatanodeInfo, String) At DataXceiver.java: [line 1163] Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859927/HDFS-9807.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux d76d37387cee 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f462e1f Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18798/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18798/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18798/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching an updated patch. This fixes the findbugs issue as well as some check style issues.

          Some check style issues are left untouched since they are consistent with the surrounding code or require considerable refactoring (parameter numbers, method length).

          Show
          ehiggs Ewan Higgs added a comment - Attaching an updated patch. This fixes the findbugs issue as well as some check style issues. Some check style issues are left untouched since they are consistent with the surrounding code or require considerable refactoring (parameter numbers, method length).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 18 new or modified test files.
          0 mvndep 1m 53s Maven dependency ordering for branch
          +1 mvninstall 13m 16s trunk passed
          +1 compile 22m 49s trunk passed
          +1 checkstyle 2m 13s trunk passed
          +1 mvnsite 2m 46s trunk passed
          +1 mvneclipse 0m 56s trunk passed
          +1 findbugs 5m 3s trunk passed
          +1 javadoc 2m 3s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 2s the patch passed
          +1 compile 16m 13s the patch passed
          +1 cc 16m 13s the patch passed
          +1 javac 16m 13s the patch passed
          -0 checkstyle 2m 14s root: The patch generated 14 new + 1537 unchanged - 53 fixed = 1551 total (was 1590)
          +1 mvnsite 2m 44s the patch passed
          +1 mvneclipse 0m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 31s the patch passed
          -1 javadoc 0m 46s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9)
          -1 unit 7m 38s hadoop-common in the patch failed.
          +1 unit 1m 5s hadoop-hdfs-client in the patch passed.
          -1 unit 71m 34s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          165m 39s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.net.TestDNS
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860126/HDFS-9807.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux cc5fc91f4399 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 59d6925
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18813/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18813/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 18 new or modified test files. 0 mvndep 1m 53s Maven dependency ordering for branch +1 mvninstall 13m 16s trunk passed +1 compile 22m 49s trunk passed +1 checkstyle 2m 13s trunk passed +1 mvnsite 2m 46s trunk passed +1 mvneclipse 0m 56s trunk passed +1 findbugs 5m 3s trunk passed +1 javadoc 2m 3s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 16m 13s the patch passed +1 cc 16m 13s the patch passed +1 javac 16m 13s the patch passed -0 checkstyle 2m 14s root: The patch generated 14 new + 1537 unchanged - 53 fixed = 1551 total (was 1590) +1 mvnsite 2m 44s the patch passed +1 mvneclipse 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 31s the patch passed -1 javadoc 0m 46s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9) -1 unit 7m 38s hadoop-common in the patch failed. +1 unit 1m 5s hadoop-hdfs-client in the patch passed. -1 unit 71m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 165m 39s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestDNS   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860126/HDFS-9807.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux cc5fc91f4399 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 59d6925 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18813/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18813/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18813/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attached a patch which fixes the javadoc issues.

          Show
          ehiggs Ewan Higgs added a comment - Attached a patch which fixes the javadoc issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 3m 54s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 18 new or modified test files.
          0 mvndep 1m 32s Maven dependency ordering for branch
          +1 mvninstall 14m 34s trunk passed
          +1 compile 18m 37s trunk passed
          +1 checkstyle 2m 34s trunk passed
          +1 mvnsite 3m 12s trunk passed
          +1 mvneclipse 1m 7s trunk passed
          +1 findbugs 5m 39s trunk passed
          +1 javadoc 2m 19s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 26s the patch passed
          +1 compile 16m 6s the patch passed
          +1 cc 16m 6s the patch passed
          +1 javac 16m 6s the patch passed
          -0 checkstyle 2m 18s root: The patch generated 14 new + 1537 unchanged - 53 fixed = 1551 total (was 1590)
          +1 mvnsite 3m 6s the patch passed
          +1 mvneclipse 1m 5s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 6m 4s the patch passed
          +1 javadoc 2m 15s the patch passed
          +1 unit 8m 12s hadoop-common in the patch passed.
          +1 unit 1m 27s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 40s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          163m 46s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862672/HDFS-9807.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux a75a960e8abf 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 443aa51
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19030/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19030/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19030/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19030/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 3m 54s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 18 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 14m 34s trunk passed +1 compile 18m 37s trunk passed +1 checkstyle 2m 34s trunk passed +1 mvnsite 3m 12s trunk passed +1 mvneclipse 1m 7s trunk passed +1 findbugs 5m 39s trunk passed +1 javadoc 2m 19s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 26s the patch passed +1 compile 16m 6s the patch passed +1 cc 16m 6s the patch passed +1 javac 16m 6s the patch passed -0 checkstyle 2m 18s root: The patch generated 14 new + 1537 unchanged - 53 fixed = 1551 total (was 1590) +1 mvnsite 3m 6s the patch passed +1 mvneclipse 1m 5s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 4s the patch passed +1 javadoc 2m 15s the patch passed +1 unit 8m 12s hadoop-common in the patch passed. +1 unit 1m 27s hadoop-hdfs-client in the patch passed. -1 unit 64m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 163m 46s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862672/HDFS-9807.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux a75a960e8abf 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 443aa51 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19030/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19030/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19030/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19030/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          HDFS-6708 has been merged so I am rebasing the patches now.

          Show
          ehiggs Ewan Higgs added a comment - HDFS-6708 has been merged so I am rebasing the patches now.
          Hide
          ehiggs Ewan Higgs added a comment -

          Attaching patch rebased now that HDFS-6708 has been merged.

          Show
          ehiggs Ewan Higgs added a comment - Attaching patch rebased now that HDFS-6708 has been merged.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 18 new or modified test files.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 14m 29s trunk passed
          +1 compile 1m 37s trunk passed
          +1 checkstyle 0m 56s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 27s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 30s the patch passed
          +1 compile 1m 51s the patch passed
          +1 cc 1m 51s the patch passed
          +1 javac 1m 51s the patch passed
          -0 checkstyle 1m 3s hadoop-hdfs-project: The patch generated 18 new + 1533 unchanged - 50 fixed = 1551 total (was 1583)
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 51s the patch passed
          +1 javadoc 1m 11s the patch passed
          +1 unit 1m 24s hadoop-hdfs-client in the patch passed.
          -1 unit 107m 57s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          147m 34s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865370/HDFS-9807.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux e9667efdb3a0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 61cda39e
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19217/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19217/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 18 new or modified test files. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 14m 29s trunk passed +1 compile 1m 37s trunk passed +1 checkstyle 0m 56s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 27s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 4s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 1m 51s the patch passed +1 cc 1m 51s the patch passed +1 javac 1m 51s the patch passed -0 checkstyle 1m 3s hadoop-hdfs-project: The patch generated 18 new + 1533 unchanged - 50 fixed = 1551 total (was 1583) +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 51s the patch passed +1 javadoc 1m 11s the patch passed +1 unit 1m 24s hadoop-hdfs-client in the patch passed. -1 unit 107m 57s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 147m 34s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865370/HDFS-9807.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux e9667efdb3a0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 61cda39e Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19217/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19217/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19217/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          virajith Virajith Jalaparti added a comment -

          Thanks for posting the modified patch Ewan Higgs. A few comments:

          1. Is there any particular reason to remove the final modifiers in DataTransferProtocol#writeBlock?
          2. As checkAccess(String[] candidateStorageIds, String[] storageIdsRequested) and checkAccess(StorageType[] candidateStorageTypes, StorageType[] storageTypesRequested) in BlockTokenSecretManager essentially perform the same function, I think instead of having a new function, it would be good to have a generic method here.
          3. It would be good to add an end-to-end test (may be using something like MiniDFSCluster to make sure that the correct storage id propagates to the VolumeChoosingPolicy.
          4. Checkstyle issues in DataXceiver, BlockReceiver need to be fixed (unused import, white space issues).

          The failed test cases seem to be known flaky tests or pass locally. So, other than the above comments, the new patch LGTM.

          Show
          virajith Virajith Jalaparti added a comment - Thanks for posting the modified patch Ewan Higgs . A few comments: Is there any particular reason to remove the final modifiers in DataTransferProtocol#writeBlock ? As checkAccess(String[] candidateStorageIds, String[] storageIdsRequested) and checkAccess(StorageType[] candidateStorageTypes, StorageType[] storageTypesRequested) in BlockTokenSecretManager essentially perform the same function, I think instead of having a new function, it would be good to have a generic method here. It would be good to add an end-to-end test (may be using something like MiniDFSCluster to make sure that the correct storage id propagates to the VolumeChoosingPolicy . Checkstyle issues in DataXceiver , BlockReceiver need to be fixed (unused import, white space issues). The failed test cases seem to be known flaky tests or pass locally. So, other than the above comments, the new patch LGTM.
          Hide
          ehiggs Ewan Higgs added a comment -

          Adding a patch that fixes check-style issues and adds a nose-tail test using MiniDFSCluster.

          Note: the test just checks that {storageId}} passed into the VolumeChoosingPolicy is part of the volume list. It doesn't test that it's the same as the one sent by the NN as part of the request. I wasn't sure of the best way to get the value from the BlockPlacementPolicy into the VolumeChoosingPolicy for comparison. If you think it's required and have a good idea of how to connect them then let me know what you think.

          Show
          ehiggs Ewan Higgs added a comment - Adding a patch that fixes check-style issues and adds a nose-tail test using MiniDFSCluster . Note: the test just checks that {storageId}} passed into the VolumeChoosingPolicy is part of the volume list. It doesn't test that it's the same as the one sent by the NN as part of the request. I wasn't sure of the best way to get the value from the BlockPlacementPolicy into the VolumeChoosingPolicy for comparison. If you think it's required and have a good idea of how to connect them then let me know what you think.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 27s Maven dependency ordering for branch
          +1 mvninstall 13m 32s trunk passed
          +1 compile 1m 27s trunk passed
          +1 checkstyle 1m 0s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 20s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 38s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 24s the patch passed
          +1 cc 1m 24s the patch passed
          +1 javac 1m 24s the patch passed
          -0 checkstyle 0m 58s hadoop-hdfs-project: The patch generated 22 new + 1536 unchanged - 50 fixed = 1558 total (was 1586)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 11s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          -1 unit 63m 30s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          98m 44s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865553/HDFS-9807.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux cad6cd8a94cc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cb672a4
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19230/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19230/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 27s Maven dependency ordering for branch +1 mvninstall 13m 32s trunk passed +1 compile 1m 27s trunk passed +1 checkstyle 1m 0s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 20s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 38s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 24s the patch passed +1 cc 1m 24s the patch passed +1 javac 1m 24s the patch passed -0 checkstyle 0m 58s hadoop-hdfs-project: The patch generated 22 new + 1536 unchanged - 50 fixed = 1558 total (was 1586) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 11s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 63m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 98m 44s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865553/HDFS-9807.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux cad6cd8a94cc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cb672a4 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19230/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19230/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19230/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          virajith Virajith Jalaparti added a comment -

          Posting a patch which is a modification of the patch posted by Ewan Higgs.

          The test is now renamed to TestNamenodeStorageDirectives from TestDataNodeStorage (also moved to the org.apache.hadoop.hdfs.server.namenode package). Also, added a new TestBlockPlacementPolicy and modified the existing TestVolumeChoosingPolicy to ensure that the storage id passed to the VolumeChoosingPolicy is exactly the same as chosen by the BlockPlacementPolicy. Also fixed some of the checkstyle issues from the earlier patch.

          Ewan Higgs please take a look to see if this makes sense. I also removed the verifyFileReplicasOnOnlyThreeStorageID as it was not being called, and changed the visibility of some of the functions in the test to private.

          Show
          virajith Virajith Jalaparti added a comment - Posting a patch which is a modification of the patch posted by Ewan Higgs . The test is now renamed to TestNamenodeStorageDirectives from TestDataNodeStorage (also moved to the org.apache.hadoop.hdfs.server.namenode package). Also, added a new TestBlockPlacementPolicy and modified the existing TestVolumeChoosingPolicy to ensure that the storage id passed to the VolumeChoosingPolicy is exactly the same as chosen by the BlockPlacementPolicy . Also fixed some of the checkstyle issues from the earlier patch. Ewan Higgs please take a look to see if this makes sense. I also removed the verifyFileReplicasOnOnlyThreeStorageID as it was not being called, and changed the visibility of some of the functions in the test to private .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 31s Maven dependency ordering for branch
          +1 mvninstall 15m 57s trunk passed
          +1 compile 1m 41s trunk passed
          +1 checkstyle 0m 58s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          -1 findbugs 1m 33s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 3s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 1m 24s the patch passed
          +1 cc 1m 24s the patch passed
          +1 javac 1m 24s the patch passed
          -0 checkstyle 0m 59s hadoop-hdfs-project: The patch generated 22 new + 1591 unchanged - 34 fixed = 1613 total (was 1625)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 11s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          -1 unit 89m 21s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          128m 53s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865630/HDFS-9807.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 910d4d8f1dc6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 19a7e94
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19239/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19239/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 31s Maven dependency ordering for branch +1 mvninstall 15m 57s trunk passed +1 compile 1m 41s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 46s trunk passed -1 findbugs 1m 33s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 3s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 1m 24s the patch passed +1 cc 1m 24s the patch passed +1 javac 1m 24s the patch passed -0 checkstyle 0m 59s hadoop-hdfs-project: The patch generated 22 new + 1591 unchanged - 34 fixed = 1613 total (was 1625) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 11s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 9s hadoop-hdfs-client in the patch passed. -1 unit 89m 21s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 128m 53s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865630/HDFS-9807.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 910d4d8f1dc6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19a7e94 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19239/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19239/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19239/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          virajith Virajith Jalaparti added a comment -

          Posting a modified patch fixing some of the checkstyle errors.

          Show
          virajith Virajith Jalaparti added a comment - Posting a modified patch fixing some of the checkstyle errors.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 14m 6s trunk passed
          +1 compile 1m 34s trunk passed
          +1 checkstyle 0m 55s trunk passed
          +1 mvnsite 1m 29s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          -1 findbugs 1m 25s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 3s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 1m 29s the patch passed
          +1 cc 1m 29s the patch passed
          +1 javac 1m 29s the patch passed
          -0 checkstyle 0m 52s hadoop-hdfs-project: The patch generated 19 new + 1590 unchanged - 34 fixed = 1609 total (was 1624)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 22s the patch passed
          +1 javadoc 0m 58s the patch passed
          +1 unit 1m 11s hadoop-hdfs-client in the patch passed.
          -1 unit 96m 5s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          132m 59s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865792/HDFS-9807.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 3592c59e7749 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 30fc580
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19250/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19250/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 14m 6s trunk passed +1 compile 1m 34s trunk passed +1 checkstyle 0m 55s trunk passed +1 mvnsite 1m 29s trunk passed +1 mvneclipse 0m 27s trunk passed -1 findbugs 1m 25s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 3s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 1m 29s the patch passed +1 cc 1m 29s the patch passed +1 javac 1m 29s the patch passed -0 checkstyle 0m 52s hadoop-hdfs-project: The patch generated 19 new + 1590 unchanged - 34 fixed = 1609 total (was 1624) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 22s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 1m 11s hadoop-hdfs-client in the patch passed. -1 unit 96m 5s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 132m 59s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.server.namenode.TestMetadataVersionOutput Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865792/HDFS-9807.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 3592c59e7749 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 30fc580 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19250/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19250/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19250/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Thanks Virajith Jalaparti! This is good stuff.

          In your solution using BlockPlacementPolicy and a static variable to record the block, I had considered such a solution but was concerned that this will break when running tests in parallel if we have multiple test cases. As it stands, we only have the one test so it should be fine; but if someone adds another test using the same BlockPlacementPolicy there is a risk that we introduce some flakiness into the testing infrastructure.

          As per your previous question about why I had removed final from arguments in a function. This is because checkstyle complained that I had added redundant final decls to a function argument, and I figured that if someone deigned to turn that warning on in checkstyle then I should fix it where I'm touching the code.

          For other people watching along: findbugs is bugged; the issues reported are not touched/introduced by this patch afaict. Same with the unit tests.

          Show
          ehiggs Ewan Higgs added a comment - Thanks Virajith Jalaparti ! This is good stuff. In your solution using BlockPlacementPolicy and a static variable to record the block, I had considered such a solution but was concerned that this will break when running tests in parallel if we have multiple test cases. As it stands, we only have the one test so it should be fine; but if someone adds another test using the same BlockPlacementPolicy there is a risk that we introduce some flakiness into the testing infrastructure. As per your previous question about why I had removed final from arguments in a function. This is because checkstyle complained that I had added redundant final decls to a function argument, and I figured that if someone deigned to turn that warning on in checkstyle then I should fix it where I'm touching the code. For other people watching along: findbugs is bugged; the issues reported are not touched/introduced by this patch afaict. Same with the unit tests.
          Hide
          virajith Virajith Jalaparti added a comment -

          was concerned that this will break when running tests in parallel if we have multiple test cases.

          I agree. It only works as long as it is used by a single test, and is used as in the current test case.

          This is because checkstyle complained that I had added redundant final decls to a function argument.

          Makes sense. May be others can comment if this needs to be changed?

          Show
          virajith Virajith Jalaparti added a comment - was concerned that this will break when running tests in parallel if we have multiple test cases. I agree. It only works as long as it is used by a single test, and is used as in the current test case. This is because checkstyle complained that I had added redundant final decls to a function argument. Makes sense. May be others can comment if this needs to be changed?
          Hide
          chris.douglas Chris Douglas added a comment -

          Makes sense. May be others can comment if this needs to be changed?

          Either way; not worth spending time pondering. Generally history is easier to read (git blame) when it's not overlaid with unrelated cosmetic changes, but that rule doesn't apply meaningfully, here.

          +1 overall. The changes to Host2NodesMap and BlockPlacementPolicyDefault are unrelated?

          I'll commit this tomorrow if there are no further comments.

          Show
          chris.douglas Chris Douglas added a comment - Makes sense. May be others can comment if this needs to be changed? Either way; not worth spending time pondering. Generally history is easier to read (git blame) when it's not overlaid with unrelated cosmetic changes, but that rule doesn't apply meaningfully, here. +1 overall. The changes to Host2NodesMap and BlockPlacementPolicyDefault are unrelated? I'll commit this tomorrow if there are no further comments.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Ewan Higgs for the working on this. It's a nice work! I take a quick look on this and just have some questions.
          For the interface of VolumeChoosingPolicy#chooseVolume, how to decide if it should respect the storageID chosen from namenode or just ignore this? A new setting to control this?
          In addition, if VolumeChoosingPolicy can respect the chosen from namenode after this patch, is that mean we should Improve the selection in choosing storage for blocks that did in DatanodeDescriptor#chooseStorage4Block. Under the currently logic, it will always return the first valid storage for block writing, I have did this improvement in HDFS-11464. I think that JIRA should be reopened and will make sense now, right?
          Thanks for the any comments.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Ewan Higgs for the working on this. It's a nice work! I take a quick look on this and just have some questions. For the interface of VolumeChoosingPolicy#chooseVolume, how to decide if it should respect the storageID chosen from namenode or just ignore this? A new setting to control this? In addition, if VolumeChoosingPolicy can respect the chosen from namenode after this patch, is that mean we should Improve the selection in choosing storage for blocks that did in DatanodeDescriptor#chooseStorage4Block . Under the currently logic, it will always return the first valid storage for block writing, I have did this improvement in HDFS-11464 . I think that JIRA should be reopened and will make sense now, right? Thanks for the any comments.
          Hide
          virajith Virajith Jalaparti added a comment - - edited

          Posted a new patch (v010) that reverts the unrelated changes to Host2NodesMap and BlockPlacementPolicyDefault as Chris Douglas pointed out. Ewan Higgs, are you ok with this? The changes to those classes in the earlier patches don't seem needed.

          Yiqun Lin I think implementing a new VolumeChoosingPolicy will be one way in which the Namenode's choice of the storageID is respected. As in the comments above (this and this ), it will be up to the VolumeChoosingPolicy to determine if the storageID from the Namenode is used. For the choice of storageID to be really respected, I think the BlockPlacementPolicy on the Namenode and the VolumeChoosingPolicy should be consistent in the way the volumes are chosen. The goal of this JIRA was to only provide the plumbing needed to propagate the storageID to the VolumeChoosingPolicy and not to implement a new VolumeChoosingPolicy. The actual policies to use can be determined separately.
          The proposal in HDFS-11464 will make sense. However, I think the change must be done in BlockPlacementPolicyDefault#chooseStorage4Block and not in DataNodeDescriptor#chooseStorage4Block. Looking at DataNodeDescriptor#chooseStorage4Block now, I think the logic implemented in DataNodeDescriptor#chooseStorage4Block can be done in BlockPlacementPolicyDefault#chooseStorage4Block itself.

          Show
          virajith Virajith Jalaparti added a comment - - edited Posted a new patch (v010) that reverts the unrelated changes to Host2NodesMap and BlockPlacementPolicyDefault as Chris Douglas pointed out. Ewan Higgs , are you ok with this? The changes to those classes in the earlier patches don't seem needed. Yiqun Lin I think implementing a new VolumeChoosingPolicy will be one way in which the Namenode's choice of the storageID is respected. As in the comments above ( this and this ), it will be up to the VolumeChoosingPolicy to determine if the storageID from the Namenode is used. For the choice of storageID to be really respected, I think the BlockPlacementPolicy on the Namenode and the VolumeChoosingPolicy should be consistent in the way the volumes are chosen. The goal of this JIRA was to only provide the plumbing needed to propagate the storageID to the VolumeChoosingPolicy and not to implement a new VolumeChoosingPolicy . The actual policies to use can be determined separately. The proposal in HDFS-11464 will make sense. However, I think the change must be done in BlockPlacementPolicyDefault#chooseStorage4Block and not in DataNodeDescriptor#chooseStorage4Block . Looking at DataNodeDescriptor#chooseStorage4Block now, I think the logic implemented in DataNodeDescriptor#chooseStorage4Block can be done in BlockPlacementPolicyDefault#chooseStorage4Block itself.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 38s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 15m 30s trunk passed
          +1 compile 1m 37s trunk passed
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 37s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          -1 findbugs 1m 35s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 9s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 1m 35s the patch passed
          +1 cc 1m 35s the patch passed
          +1 javac 1m 35s the patch passed
          -0 checkstyle 0m 59s hadoop-hdfs-project: The patch generated 19 new + 1550 unchanged - 34 fixed = 1569 total (was 1584)
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 18s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 1m 17s hadoop-hdfs-client in the patch passed.
          -1 unit 67m 53s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          107m 34s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-9807
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866462/HDFS-9807.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 064c0e8a2458 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c2a52ef
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19313/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19313/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 38s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 15m 30s trunk passed +1 compile 1m 37s trunk passed +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 37s trunk passed +1 mvneclipse 0m 32s trunk passed -1 findbugs 1m 35s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 53s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 9s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 1m 35s the patch passed +1 cc 1m 35s the patch passed +1 javac 1m 35s the patch passed -0 checkstyle 0m 59s hadoop-hdfs-project: The patch generated 19 new + 1550 unchanged - 34 fixed = 1569 total (was 1584) +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 18s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 1m 17s hadoop-hdfs-client in the patch passed. -1 unit 67m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 107m 34s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-9807 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866462/HDFS-9807.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 064c0e8a2458 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c2a52ef Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19313/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19313/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19313/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ehiggs Ewan Higgs added a comment -

          Posted a new patch (v010) that reverts the unrelated changes to Host2NodesMap and BlockPlacementPolicyDefault as Chris Douglas pointed out. Ewan Higgs, are you ok with this? The changes to those classes in the earlier patches don't seem needed.

          Yes, that's fine. The reason Host2NodesMap was public was because I had originally made the test you added with BlockPlacementPolicy. That requires Host2NodesMap in the initialize function. I like your solution of using BlockPlacementPolicyDefault better.

          Yiqun Lin, Virajith Jalaparti's explanation is spot on. Most notably:

          The goal of this JIRA was to only provide the plumbing needed to propagate the storageID to the VolumeChoosingPolicy and not to implement a new VolumeChoosingPolicy. The actual policies to use can be determined separately.

          Show
          ehiggs Ewan Higgs added a comment - Posted a new patch (v010) that reverts the unrelated changes to Host2NodesMap and BlockPlacementPolicyDefault as Chris Douglas pointed out. Ewan Higgs, are you ok with this? The changes to those classes in the earlier patches don't seem needed. Yes, that's fine. The reason Host2NodesMap was public was because I had originally made the test you added with BlockPlacementPolicy . That requires Host2NodesMap in the initialize function. I like your solution of using BlockPlacementPolicyDefault better. Yiqun Lin , Virajith Jalaparti 's explanation is spot on. Most notably: The goal of this JIRA was to only provide the plumbing needed to propagate the storageID to the VolumeChoosingPolicy and not to implement a new VolumeChoosingPolicy. The actual policies to use can be determined separately.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Ewan Higgs and Virajith Jalaparti for the concrete explanation. The comments make sense to me. .

          Show
          linyiqun Yiqun Lin added a comment - Thanks Ewan Higgs and Virajith Jalaparti for the concrete explanation. The comments make sense to me. .
          Hide
          chris.douglas Chris Douglas added a comment -

          I committed this. Thanks Ewan

          Show
          chris.douglas Chris Douglas added a comment - I committed this. Thanks Ewan
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11691/)
          HDFS-9807. Add an optional StorageID to writes. Contributed by Ewan (cdouglas: rev a3954ccab148bddc290cb96528e63ff19799bcc9)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestAvailableSpaceVolumeChoosingPolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/token/block/TestBlockToken.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeStorageDirectives.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockPoolTokenSecretManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataXceiverLazyPersistHint.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/KeyManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedWriter.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/AvailableSpaceVolumeChoosingPolicy.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructionInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockWriter.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11691/ ) HDFS-9807 . Add an optional StorageID to writes. Contributed by Ewan (cdouglas: rev a3954ccab148bddc290cb96528e63ff19799bcc9) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestAvailableSpaceVolumeChoosingPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/token/block/TestBlockToken.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeStorageDirectives.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockPoolTokenSecretManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataXceiverLazyPersistHint.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/KeyManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedWriter.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/AvailableSpaceVolumeChoosingPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructionInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockWriter.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
          Hide
          andrew.wang Andrew Wang added a comment -

          I believe this (or HDFS-6708) broke backwards compatibility with older clients. I'm running a 2.6.0-ish client against a 3.0.0-alpha4-ish DN, and am seeing this in the DN log:

          2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block token verification failed: op=WRITE_BLOCK, remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with StorageIDs []
          

          The client then blacklists each DN when it gets this error, until it runs out of DNs. Dunno if this retry behavior is fixed in newer clients; InvalidBlockTokenExceptions are kind of fatal.

          What is the expected behavior here?

          Show
          andrew.wang Andrew Wang added a comment - I believe this (or HDFS-6708 ) broke backwards compatibility with older clients. I'm running a 2.6.0-ish client against a 3.0.0-alpha4-ish DN, and am seeing this in the DN log: 2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block token verification failed: op=WRITE_BLOCK, remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with StorageIDs [] The client then blacklists each DN when it gets this error, until it runs out of DNs. Dunno if this retry behavior is fixed in newer clients; InvalidBlockTokenExceptions are kind of fatal. What is the expected behavior here?
          Hide
          chris.douglas Chris Douglas added a comment -

          What is the expected behavior here?

          Not that.

          Looking into it.

          Show
          chris.douglas Chris Douglas added a comment - What is the expected behavior here? Not that. Looking into it.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks Chris. I've filed HDFS-11956 for release tracking purposes, we can do any additional code changes over there.

          Show
          andrew.wang Andrew Wang added a comment - Thanks Chris. I've filed HDFS-11956 for release tracking purposes, we can do any additional code changes over there.

            People

            • Assignee:
              ehiggs Ewan Higgs
              Reporter:
              chris.douglas Chris Douglas
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development