Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1362 Provide volume management functionality for DataNode
  3. HDFS-7035

Make adding a new data directory to the DataNode an atomic operation and improve error handling

    Details

    • Target Version/s:

      Description

      It refactors DataStorage and BlockPoolSliceStorage to reduce the duplicate code and supports atomic adding volume operations. Also it parallels loading data volume operation: each thread loads one volume.

      1. HDFS-7035.000.combo.patch
        99 kB
        Lei (Eddy) Xu
      2. HDFS-7035.000.patch
        40 kB
        Lei (Eddy) Xu
      3. HDFS-7035.001.combo.patch
        99 kB
        Lei (Eddy) Xu
      4. HDFS-7035.001.patch
        40 kB
        Lei (Eddy) Xu
      5. HDFS-7035.002.patch
        41 kB
        Lei (Eddy) Xu
      6. HDFS-7035.003.patch
        41 kB
        Lei (Eddy) Xu
      7. HDFS-7035.003.patch
        40 kB
        Lei (Eddy) Xu
      8. HDFS-7035.004.patch
        72 kB
        Lei (Eddy) Xu
      9. HDFS-7035.005.patch
        73 kB
        Lei (Eddy) Xu
      10. HDFS-7035.007.patch
        58 kB
        Lei (Eddy) Xu
      11. HDFS-7035.008.patch
        58 kB
        Lei (Eddy) Xu
      12. HDFS-7035.009.patch
        58 kB
        Lei (Eddy) Xu
      13. HDFS-7035.010.patch
        96 kB
        Lei (Eddy) Xu
      14. HDFS-7035.010.patch
        96 kB
        Lei (Eddy) Xu
      15. HDFS-7035.011.patch
        58 kB
        Lei (Eddy) Xu
      16. HDFS-7035.012.patch
        64 kB
        Lei (Eddy) Xu
      17. HDFS-7035.013.patch
        66 kB
        Lei (Eddy) Xu
      18. HDFS-7035.014.patch
        66 kB
        Lei (Eddy) Xu
      19. HDFS-7035.015.patch
        66 kB
        Lei (Eddy) Xu
      20. HDFS-7035.016.patch
        66 kB
        Lei (Eddy) Xu

        Issue Links

          Activity

          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          This patch extract the common logic of recoverTransitionRead from DataStorage and BlockSlicePoolStorage to a new base class DatanodeStorageBase.

          Show
          eddyxu Lei (Eddy) Xu added a comment - This patch extract the common logic of recoverTransitionRead from DataStorage and BlockSlicePoolStorage to a new base class DatanodeStorageBase .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667494/HDFS-7035.000.combo.patch
          against trunk revision 3e8f353.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7974//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667494/HDFS-7035.000.combo.patch against trunk revision 3e8f353. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7974//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Update patch to resolve conflicts against trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Update patch to resolve conflicts against trunk.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667501/HDFS-7035.001.combo.patch
          against trunk revision 3e8f353.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestLeaseRecovery
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          org.apache.hadoop.hdfs.TestSafeMode
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          org.apache.hadoop.hdfs.TestDatanodeRegistration
          org.apache.hadoop.hdfs.TestHFlush
          org.apache.hadoop.hdfs.TestSetTimes
          org.apache.hadoop.hdfs.TestPersistBlocks
          org.apache.hadoop.hdfs.TestEncryptedTransfer
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.net.TestNetworkTopology
          org.apache.hadoop.hdfs.server.namenode.TestFSImage
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.tools.TestDFSAdmin
          org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
          org.apache.hadoop.hdfs.TestDecommission
          org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.fs.permission.TestStickyBit
          org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
          org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag
          org.apache.hadoop.hdfs.web.TestWebHDFSAcl
          org.apache.hadoop.hdfs.TestHDFSTrash
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.TestFileCreation
          org.apache.hadoop.hdfs.TestDataTransferKeepalive
          org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
          org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl
          org.apache.hadoop.hdfs.TestRenameWhileOpen
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr
          org.apache.hadoop.fs.TestHDFSFileContextMainOperations
          org.apache.hadoop.hdfs.web.TestWebHDFSXAttr
          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl
          org.apache.hadoop.hdfs.server.namenode.TestStartup
          org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
          org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7976//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7976//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7976//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667501/HDFS-7035.001.combo.patch against trunk revision 3e8f353. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.TestDatanodeRegistration org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.TestPersistBlocks org.apache.hadoop.hdfs.TestEncryptedTransfer org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.net.TestNetworkTopology org.apache.hadoop.hdfs.server.namenode.TestFSImage org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.tools.TestDFSAdmin org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.fs.permission.TestStickyBit org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag org.apache.hadoop.hdfs.web.TestWebHDFSAcl org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.hdfs.TestDataTransferKeepalive org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr org.apache.hadoop.fs.TestHDFSFileContextMainOperations org.apache.hadoop.hdfs.web.TestWebHDFSXAttr org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl org.apache.hadoop.hdfs.server.namenode.TestStartup org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7976//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7976//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7976//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          I will update the patch to address those failures

          Show
          eddyxu Lei (Eddy) Xu added a comment - I will update the patch to address those failures
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Rework against trunk directly.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Rework against trunk directly.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668197/HDFS-7035.002.patch
          against trunk revision 6c08339.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8002//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8002//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668197/HDFS-7035.002.patch against trunk revision 6c08339. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8002//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8002//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          TestDataNodeVolumeFailureToleration is a related failure. I will address the failure soon.

          Show
          eddyxu Lei (Eddy) Xu added a comment - TestDataNodeVolumeFailureToleration is a related failure. I will address the failure soon.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Lei, could you please provide a summary or overview of the changes? Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Lei, could you please provide a summary or overview of the changes? Thanks.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Arpit Agarwal Thanks very much for looking into this.

          This patch serves three purposes:

          1. For the volume management functionality, it refactors the loading storage directories, so that if there is a failure during the loading a storage directory, the partial loaded metadata belonging to this storage directory is completed removed from DataStorage and BlockStorage, all partial loaded directories are unlocked, and etc. It is basically for more gracefully to handle the failures of loading volumes when DataNode is running.
          2. It eliminates the duplicated logic in DataStorage and BlockSlicePoolStorage to simplify the failure recovery logic needed to be handle from DataNode.
          3. Also it changes the order to add storage directories. In the original code, all storage directories for one namespace is loading in batch. This behavior is changed to add one storage directory for one namespace at a time, also for easier failure handling for upper level (e.g., HDFS-6727) in the hot swap volume scenarios.

          Arpit Agarwal do the above answer your questions? It would be great to if you could give me some suggestions on this.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Arpit Agarwal Thanks very much for looking into this. This patch serves three purposes: For the volume management functionality, it refactors the loading storage directories, so that if there is a failure during the loading a storage directory , the partial loaded metadata belonging to this storage directory is completed removed from DataStorage and BlockStorage , all partial loaded directories are unlocked, and etc. It is basically for more gracefully to handle the failures of loading volumes when DataNode is running. It eliminates the duplicated logic in DataStorage and BlockSlicePoolStorage to simplify the failure recovery logic needed to be handle from DataNode . Also it changes the order to add storage directories. In the original code, all storage directories for one namespace is loading in batch. This behavior is changed to add one storage directory for one namespace at a time, also for easier failure handling for upper level (e.g., HDFS-6727 ) in the hot swap volume scenarios. Arpit Agarwal do the above answer your questions? It would be great to if you could give me some suggestions on this.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Update patch to fix TestDataNodeVolumeFailureToleration. It fixes the bug that BlockSlicePoolStorage locks storage directory, which was not locked in original code.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Update patch to fix TestDataNodeVolumeFailureToleration . It fixes the bug that BlockSlicePoolStorage locks storage directory , which was not locked in original code.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668501/HDFS-7035.003.patch
          against trunk revision a0ad975.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestDataStorage
          org.apache.hadoop.tracing.TestTracing
          org.apache.hadoop.hdfs.TestCrcCorruption

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8012//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8012//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668501/HDFS-7035.003.patch against trunk revision a0ad975. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestDataStorage org.apache.hadoop.tracing.TestTracing org.apache.hadoop.hdfs.TestCrcCorruption +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8012//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8012//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Update patch to address failures.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Update patch to address failures.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668558/HDFS-7035.003.patch
          against trunk revision 98588cf.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestDataTransferKeepalive

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8016//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8016//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668558/HDFS-7035.003.patch against trunk revision 98588cf. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestBPOfferService org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestDataTransferKeepalive +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8016//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8016//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, all

          I've run TestHDFSTrash,TestDistributedFileSystem,TestFileCreation,TestDataStorage,TestDataNodeRollingUpgrade,TestDatanodeStorageBase,TestBlockPoolSliceStorage,TestBPOfferService,TestFileAppend4,TestDataTransferKeepalive locally without failures.

          I think that the above failures are not related to this patch.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, all I've run TestHDFSTrash,TestDistributedFileSystem,TestFileCreation,TestDataStorage,TestDataNodeRollingUpgrade,TestDatanodeStorageBase,TestBlockPoolSliceStorage,TestBPOfferService,TestFileAppend4,TestDataTransferKeepalive locally without failures. I think that the above failures are not related to this patch.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Make addVolume() become an atomic operation. The volume metadata in DataStorage and FsDataset is first load into a local copy. After all I/O finishes, if there is nothing failed, then the DataNode commits the loaded volume metadata to DataStorage and FsDataset respectively. Therefore, if there is any error happened during loading a volume, the metadata belonging to this volume will not be visible to the service.

          Also it captures the error message for IOExceptions in DataStorage#removeVolumes().

          Show
          eddyxu Lei (Eddy) Xu added a comment - Make addVolume() become an atomic operation. The volume metadata in DataStorage and FsDataset is first load into a local copy. After all I/O finishes, if there is nothing failed, then the DataNode commits the loaded volume metadata to DataStorage and FsDataset respectively. Therefore, if there is any error happened during loading a volume, the metadata belonging to this volume will not be visible to the service. Also it captures the error message for IOExceptions in DataStorage#removeVolumes() .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12670799/HDFS-7035.004.patch
          against trunk revision 3dc28e2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          -1 javac. The applied patch generated 1265 javac compiler warnings (more than the trunk's current 1264 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestParallelUnixDomainRead

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8170//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8170//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8170//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670799/HDFS-7035.004.patch against trunk revision 3dc28e2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. -1 javac . The applied patch generated 1265 javac compiler warnings (more than the trunk's current 1264 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestParallelUnixDomainRead +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8170//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8170//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8170//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Update patch to clean up conflicts with trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Update patch to clean up conflicts with trunk .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12672126/HDFS-7035.005.patch
          against trunk revision 9582a50.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          -1 javac. The applied patch generated 1265 javac compiler warnings (more than the trunk's current 1264 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
          org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12672126/HDFS-7035.005.patch against trunk revision 9582a50. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. -1 javac . The applied patch generated 1265 javac compiler warnings (more than the trunk's current 1264 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS org.apache.hadoop.hdfs.server.datanode.TestBPOfferService org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8273//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for working on this.

          The patch needs to be updated to rebase on trunk. It didn't apply for me.

          Can you try using a utility class rather than creating a new layer in the Storage inheritance hierarchy? Implementation inheritance can get confusing. Perhaps a util class would allow BlockPoolSliceStorage and DataStorage to share code just as well, without adding more up and down calls.

                    exceptions.add(service.submit(new Callable<IOException>() {
                      \@Override
                      public IOException call() throws Exception {
                        try {
          

          Can you update the JIRA description to point out that we're also parallelizing loading volumes?

          It might be nice to catch the non-IOExceptions here and wrap them in an IOE. Ideally, there would be no non-IOEs, but if there are (because of a bug or runtime issue), we'd like to handle them.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for working on this. The patch needs to be updated to rebase on trunk. It didn't apply for me. Can you try using a utility class rather than creating a new layer in the Storage inheritance hierarchy? Implementation inheritance can get confusing. Perhaps a util class would allow BlockPoolSliceStorage and DataStorage to share code just as well, without adding more up and down calls. exceptions.add(service.submit( new Callable<IOException>() { \@Override public IOException call() throws Exception { try { Can you update the JIRA description to point out that we're also parallelizing loading volumes? It might be nice to catch the non-IOExceptions here and wrap them in an IOE. Ideally, there would be no non-IOEs, but if there are (because of a bug or runtime issue), we'd like to handle them.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Colin P. McCabe Thanks for the reviews. I have removed the new class hierarchical in this patch.

          I clear the IOException catching code, there is no other Exception, so I removed the "throws Exception" from the call() signature.

          Could you take another look?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Colin P. McCabe Thanks for the reviews. I have removed the new class hierarchical in this patch. I clear the IOException catching code, there is no other Exception, so I removed the "throws Exception" from the call() signature. Could you take another look?
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Update the patch to fix a bug in error message construction.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Update the patch to fix a bug in error message construction.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12677002/HDFS-7035.008.patch
          against trunk revision a52eb4b.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8530//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677002/HDFS-7035.008.patch against trunk revision a52eb4b. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8530//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Update patch to fix conflicts to trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Update patch to fix conflicts to trunk.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676988/HDFS-7035.007.patch
          against trunk revision 501e49f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.TestDFSAdmin

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestDistributedFileSystem

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8528//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8528//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8528//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676988/HDFS-7035.007.patch against trunk revision 501e49f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.TestDFSAdmin The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDistributedFileSystem +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8528//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8528//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8528//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          I am working on addressing the fix bug reports and related failure case (TestDistributedFileSystem)

          Show
          eddyxu Lei (Eddy) Xu added a comment - I am working on addressing the fix bug reports and related failure case (TestDistributedFileSystem)
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12677004/HDFS-7035.009.patch
          against trunk revision a52eb4b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1265 javac compiler warnings (more than the trunk's current 1264 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.viewfs.TestViewFsWithAcls
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestParallelRead

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8531//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8531//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8531//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677004/HDFS-7035.009.patch against trunk revision a52eb4b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1265 javac compiler warnings (more than the trunk's current 1264 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.viewfs.TestViewFsWithAcls org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestParallelRead +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8531//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8531//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8531//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Updated the patch to address the fingbugs reports. The javac warnings seems not related, since I could not find the changed java files in the javac warning reports. Tests failuares are also not related.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Updated the patch to address the fingbugs reports. The javac warnings seems not related, since I could not find the changed java files in the javac warning reports. Tests failuares are also not related.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12677124/HDFS-7035.010.patch
          against trunk revision c51e53d.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8541//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677124/HDFS-7035.010.patch against trunk revision c51e53d. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8541//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          The previous build is on H9. I have verified on my local machine that this patch can be applied to trunk. I reupload this patch to trigger another build.

          Show
          eddyxu Lei (Eddy) Xu added a comment - The previous build is on H9. I have verified on my local machine that this patch can be applied to trunk. I reupload this patch to trigger another build.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12677149/HDFS-7035.010.patch
          against trunk revision 65d95b1.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8542//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677149/HDFS-7035.010.patch against trunk revision 65d95b1. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8542//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Upload the patch that is updated against the newest trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Upload the patch that is updated against the newest trunk.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12677341/HDFS-7035.011.patch
          against trunk revision baf794d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8550//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8550//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677341/HDFS-7035.011.patch against trunk revision baf794d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 eclipse:eclipse . The patch failed to build with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8550//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8550//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -
          ======================================================================
          ======================================================================
              Running mvn eclipse:eclipse.
          ======================================================================
          ======================================================================
          
          
          /home/jenkins/tools/maven/latest/bin/mvn eclipse:eclipse -DHadoopPatchProcess > /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocess/patchEclipseOutput.txt 2>&1
          /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/test-patch.sh: line 692: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocess/patchEclipseOutput.txt: No such file or directory
          
          

          The eclipse:eclipse error seems not related. It looks like that the failure is resulted by the Jenkins' building script.

          Show
          eddyxu Lei (Eddy) Xu added a comment - ====================================================================== ====================================================================== Running mvn eclipse:eclipse. ====================================================================== ====================================================================== /home/jenkins/tools/maven/latest/bin/mvn eclipse:eclipse -DHadoopPatchProcess > /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocess/patchEclipseOutput.txt 2>&1 /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/test-patch.sh: line 692: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocess/patchEclipseOutput.txt: No such file or directory The eclipse:eclipse error seems not related. It looks like that the failure is resulted by the Jenkins' building script.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          I do sort of see the elegance of having the StagedVolume approach. But I think it would be a lot easier to simply:

          • step #1: add the new DataStorages
          • step #2: for all the DataStorages that we successfully added, call FsDatasetSpi#addVolumes
          • step #3: for all the volumes that failed in step #2, remove the DataStorages

          Then we don't have to change the FsDatasetSpi interface again. [We changed it once for this in HDFS-6727.] We don't have to create a new callback interface with commit, and the patch gets a lot smaller.

          I can see there may be some benefit to parallelizing searching the new directory for replica files, but this seems like an optimization we should do in a separate JIRA. Of course, we can do that inside FsDatasetSpi without changing the interface, the same way you're doing it here... with Futures. My impression is that the amount of work we do in DataStorage is really small, so there isn't any need to parallelize the DataStorage stuff, only the scanning of the replicas.

          I'm having trouble figuring out the locking for the callback-based approach here. Right now we're holding the dn lock when calling commit... but I don't see why. Locking would be clearer if we just avoided callbacks and went for the straight-line approach, I think. What do you think?

          506    int numOldDataDirs = 0;
          507	    synchronized (this) {
          508	      numOldDataDirs = dataDirs.size();
          509	      dataDirs = locations;
          510	    }
          

          Do we still need to set DataNode#dataDirs here? And I don't see where we change this back on failure, if we fail.

          Show
          cmccabe Colin P. McCabe added a comment - - edited I do sort of see the elegance of having the StagedVolume approach. But I think it would be a lot easier to simply: step #1: add the new DataStorages step #2: for all the DataStorages that we successfully added, call FsDatasetSpi#addVolumes step #3: for all the volumes that failed in step #2, remove the DataStorages Then we don't have to change the FsDatasetSpi interface again. [We changed it once for this in HDFS-6727.] We don't have to create a new callback interface with commit , and the patch gets a lot smaller. I can see there may be some benefit to parallelizing searching the new directory for replica files, but this seems like an optimization we should do in a separate JIRA. Of course, we can do that inside FsDatasetSpi without changing the interface, the same way you're doing it here... with Futures. My impression is that the amount of work we do in DataStorage is really small, so there isn't any need to parallelize the DataStorage stuff, only the scanning of the replicas. I'm having trouble figuring out the locking for the callback-based approach here. Right now we're holding the dn lock when calling commit ... but I don't see why. Locking would be clearer if we just avoided callbacks and went for the straight-line approach, I think. What do you think? 506 int numOldDataDirs = 0; 507 synchronized ( this ) { 508 numOldDataDirs = dataDirs.size(); 509 dataDirs = locations; 510 } Do we still need to set DataNode#dataDirs here? And I don't see where we change this back on failure, if we fail.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Colin P. McCabe Thanks for your detailed comments!

          In this updated patch, I have made the following major changes:

          • Merge the changes from HDFS-7173 so that this patch can be more self-contained.
          • Remove StagedAddVolume interface and rename DataStorage#DataStorageAddedVolume to DataStorage#VolumeBuilder.
          • Removed FsDatasetImpl#StagedAddVolume
          • Moved the logic of calling DataStorage#addVolume from DataNode#refreshVolume to FsDatasetImpl#addVolume.
          • Change DataNode#refreshVolume to syncrhonized function so that there will be no starting-up / shutting-down DN activities.
          • Move DataNode#conf and DataNode#dataDir recovery into the finally segment.
          Show
          eddyxu Lei (Eddy) Xu added a comment - Colin P. McCabe Thanks for your detailed comments! In this updated patch, I have made the following major changes: Merge the changes from HDFS-7173 so that this patch can be more self-contained. Remove StagedAddVolume interface and rename DataStorage#DataStorageAddedVolume to DataStorage#VolumeBuilder . Removed FsDatasetImpl#StagedAddVolume Moved the logic of calling DataStorage#addVolume from DataNode#refreshVolume to FsDatasetImpl#addVolume . Change DataNode#refreshVolume to syncrhonized function so that there will be no starting-up / shutting-down DN activities. Move DataNode#conf and DataNode#dataDir recovery into the finally segment.
          Hide
          cmccabe Colin P. McCabe added a comment -

          This looks good. Thanks, Eddy. Comments below.

            private synchronized void refreshVolumes(String newVolumes) throws IOException {
                conf.set(DFS_DATANODE_DATA_DIR_KEY, newVolumes);
          

          What's the purpose of setting this at the beginning of the function? At the end of the function we set it to the actual volumes that got added (plus the existing). It seems like it only needs to be set once?

          Also, we should add some JavaDoc stating that even if an IOException is thrown from this function, some new volumes may have been successfully added.

          LOG.info("Analyzed volume - " + dir + ", StorageType: " + storageType);
          

          Should say "added volume"? Since this is at the end of addVolume.

          updateReplicaUnderRecovery: can we avoid changing the whitespace here? It's distracting

          SimulatedFSDataset#addVolume: need an Override annotation here. Findbugs or something will probably complain.

          1115      public void write(StorageDirectory sd) throws IOException {
          1116	    this.layoutVersion = getServiceLayoutVersion();
          1117	    writeProperties(sd);
          1118	  }
          

          I realize that you modelled this on Storage#writeAll. But I find this to be a weird (and weirdly named) API. It's a function named "write", that updates the layoutVersion? And then writes just the properties file?

          I think we should have an API named setServiceLayoutVersion, and then just do setServiceLayoutVersion(getServiceLayoutVersion()). Better yet, rename getServiceLayoutVersion to getLatestServiceLayoutVersion, since that's really what it's doing.

          Then we could just call:

          storage.setServiceLayoutVersion(getLatestServiceLayoutVersion())
          storage.writeProperties(sd)
          

          and it would be obvious what was going on.

          Show
          cmccabe Colin P. McCabe added a comment - This looks good. Thanks, Eddy. Comments below. private synchronized void refreshVolumes( String newVolumes) throws IOException { conf.set(DFS_DATANODE_DATA_DIR_KEY, newVolumes); What's the purpose of setting this at the beginning of the function? At the end of the function we set it to the actual volumes that got added (plus the existing). It seems like it only needs to be set once? Also, we should add some JavaDoc stating that even if an IOException is thrown from this function, some new volumes may have been successfully added. LOG.info( "Analyzed volume - " + dir + ", StorageType: " + storageType); Should say "added volume"? Since this is at the end of addVolume. updateReplicaUnderRecovery : can we avoid changing the whitespace here? It's distracting SimulatedFSDataset#addVolume : need an Override annotation here. Findbugs or something will probably complain. 1115 public void write(StorageDirectory sd) throws IOException { 1116 this .layoutVersion = getServiceLayoutVersion(); 1117 writeProperties(sd); 1118 } I realize that you modelled this on Storage#writeAll . But I find this to be a weird (and weirdly named) API. It's a function named "write", that updates the layoutVersion? And then writes just the properties file? I think we should have an API named setServiceLayoutVersion, and then just do setServiceLayoutVersion(getServiceLayoutVersion()). Better yet, rename getServiceLayoutVersion to getLatestServiceLayoutVersion, since that's really what it's doing. Then we could just call: storage.setServiceLayoutVersion(getLatestServiceLayoutVersion()) storage.writeProperties(sd) and it would be obvious what was going on.
          Hide
          cmccabe Colin P. McCabe added a comment -

          P.S. I like the VolumeBuilder concept. Can you add some JavaDoc about the usage? I am also curious what happens when we fail midway through. I can see that DataStorage#prepareVolume adds the volume to DataStorage#bpStorageMap, is there anywhere where we remove it if the addition fails? Perhaps we need an abort function in the Builder?

          Show
          cmccabe Colin P. McCabe added a comment - P.S. I like the VolumeBuilder concept. Can you add some JavaDoc about the usage? I am also curious what happens when we fail midway through. I can see that DataStorage#prepareVolume adds the volume to DataStorage#bpStorageMap , is there anywhere where we remove it if the addition fails? Perhaps we need an abort function in the Builder?
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Colin P. McCabe Thanks for your quick reviews.

          I have updates the patch based on your comments. conf is only updated in the end of refreshVolumes.

          I am also curious what happens when we fail midway through. I can see that DataStorage#prepareVolume adds the volume to DataStorage#bpStorageMap, is there anywhere where we remove it if the addition fails?

          I have also added comments here. Basically, refreshVolume does not change the states of BlockPoolSliceStorage except updating a few member with constant values.

          Would you give another look? Thanks!

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Colin P. McCabe Thanks for your quick reviews. I have updates the patch based on your comments. conf is only updated in the end of refreshVolumes . I am also curious what happens when we fail midway through. I can see that DataStorage#prepareVolume adds the volume to DataStorage#bpStorageMap, is there anywhere where we remove it if the addition fails? I have also added comments here. Basically, refreshVolume does not change the states of BlockPoolSliceStorage except updating a few member with constant values. Would you give another look? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678017/HDFS-7035.012.patch
          against trunk revision d33e07d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 javac. The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1267 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8590//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8590//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8590//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678017/HDFS-7035.012.patch against trunk revision d33e07d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 javac . The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1267 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8590//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8590//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8590//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Storage.java: remove unnecessary whitespace change

          129	  // Expose visibility for VolumeBuilder#commit().
          130	  public void addStorageDir(StorageDirectory sd) {
          131	    super.addStorageDir(sd);
          132	  }
          

          This isn't needed because VolumeBuilder is in the same Java package as Storage, so it can just call the parent method directly.

              /** The unchanged locations that exist in the old configuration. */
          

          Should be "existed in the old configuration"

                builder.addBpStorageDirtectories(
          

          Should be "directories" not "dirtectories"

          167	      // 2. Do transitions
          168	      // Each storage directory is treated individually.
          169	      // During startup some of them can upgrade or roll back
          170	      // while others could be up-to-date for the regular startup.
          171	      doTransition(datanode, sd, nsInfo, startOpt);
          172	      assert getCTime() == nsInfo.getCTime()
          173	          : "Data-node and name-node CTimes must be the same.";
          

          This should be throwing an IOE, not an assert. Otherwise we're bringing down the DataNode because someone tried to add a storage directory that wasn't valid... not good.

          327	      // bpStorage does not add loaded volume immediately. The volume will be
          328	      // added when calling builder.build() later. However, several
          329	      // members (e.g., Storage#layoutVersion, Storage#cTime will be updated
          330	      // in BlockPoolSliceStorage#format() and
          331	      // BlockPoolSliceStorage#loadStorageDirectory. But since these values are
          332	      // considered constant during the DataNode execution, we do not revert the
          333	      // the changes on such members.
          

          I think this comment belongs in the JavaDoc for the function. I also feel like the current form of the comment is somewhat confusing. I would say something like "prepareVolume creates a builder which can be used to add to the volume. If the volume cannot be added, it is OK to discard the builder later."

          removeVolumes: can you document in the JavaDoc for this function that even when the IOE is thrown, the volumes are still removed?

          +1 once these are addressed.

          Show
          cmccabe Colin P. McCabe added a comment - Storage.java : remove unnecessary whitespace change 129 // Expose visibility for VolumeBuilder#commit(). 130 public void addStorageDir(StorageDirectory sd) { 131 super .addStorageDir(sd); 132 } This isn't needed because VolumeBuilder is in the same Java package as Storage, so it can just call the parent method directly. /** The unchanged locations that exist in the old configuration. */ Should be "existed in the old configuration" builder.addBpStorageDirtectories( Should be "directories" not "dirtectories" 167 // 2. Do transitions 168 // Each storage directory is treated individually. 169 // During startup some of them can upgrade or roll back 170 // while others could be up-to-date for the regular startup. 171 doTransition(datanode, sd, nsInfo, startOpt); 172 assert getCTime() == nsInfo.getCTime() 173 : "Data-node and name-node CTimes must be the same." ; This should be throwing an IOE, not an assert. Otherwise we're bringing down the DataNode because someone tried to add a storage directory that wasn't valid... not good. 327 // bpStorage does not add loaded volume immediately. The volume will be 328 // added when calling builder.build() later. However, several 329 // members (e.g., Storage#layoutVersion, Storage#cTime will be updated 330 // in BlockPoolSliceStorage#format() and 331 // BlockPoolSliceStorage#loadStorageDirectory. But since these values are 332 // considered constant during the DataNode execution, we do not revert the 333 // the changes on such members. I think this comment belongs in the JavaDoc for the function. I also feel like the current form of the comment is somewhat confusing. I would say something like "prepareVolume creates a builder which can be used to add to the volume. If the volume cannot be added, it is OK to discard the builder later." removeVolumes: can you document in the JavaDoc for this function that even when the IOE is thrown, the volumes are still removed? +1 once these are addressed.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Colin P. McCabe I made changes based on your comments. Thanks!

          this isn't needed because VolumeBuilder is in the same Java package as Storage.

          This function is BlockPoolSliceStorage#addStorageDir, which is called by DataStorage#VolumeBuilder. I could not use protected or project visibility here.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Colin P. McCabe I made changes based on your comments. Thanks! this isn't needed because VolumeBuilder is in the same Java package as Storage. This function is BlockPoolSliceStorage#addStorageDir , which is called by DataStorage#VolumeBuilder . I could not use protected or project visibility here.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678067/HDFS-7035.013.patch
          against trunk revision 6f5f604.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 javac. The applied patch generated 1272 javac compiler warnings (more than the trunk's current 1267 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8594//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8594//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8594//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678067/HDFS-7035.013.patch against trunk revision 6f5f604. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 javac . The applied patch generated 1272 javac compiler warnings (more than the trunk's current 1267 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8594//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8594//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8594//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678104/HDFS-7035.014.patch
          against trunk revision 2a6be65.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 javac. The applied patch generated 1266 javac compiler warnings (more than the trunk's current 1265 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
          org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
          org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes
          org.apache.hadoop.hdfs.TestEncryptedTransfer
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
          org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream
          org.apache.hadoop.hdfs.TestSnapshotCommands
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
          org.apache.hadoop.hdfs.TestRead
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
          org.apache.hadoop.hdfs.TestBlocksScheduledCounter
          org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
          org.apache.hadoop.hdfs.TestDFSPermission
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.server.namenode.TestStartup
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithXAttr
          org.apache.hadoop.hdfs.TestDFSClientFailover
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
          org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
          org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          org.apache.hadoop.hdfs.TestLeaseRecovery2
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
          org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader
          org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation
          org.apache.hadoop.hdfs.web.TestHttpsFileSystem
          org.apache.hadoop.hdfs.server.namenode.TestFSDirectory
          org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
          org.apache.hadoop.hdfs.TestDFSOutputStream
          org.apache.hadoop.hdfs.TestSetTimes
          org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
          org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          org.apache.hadoop.hdfs.TestDatanodeDeath
          org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
          org.apache.hadoop.hdfs.TestDFSRollback
          org.apache.hadoop.hdfs.TestClientBlockVerification
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit
          org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
          org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
          org.apache.hadoop.hdfs.TestFileCreationEmpty
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
          org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
          org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
          org.apache.hadoop.hdfs.server.datanode.TestDiskError
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeRpcServer
          org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes
          org.apache.hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd
          org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
          org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          org.apache.hadoop.hdfs.TestSetrepIncreasing
          org.apache.hadoop.hdfs.TestReadWhileWriting
          org.apache.hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy
          org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          org.apache.hadoop.hdfs.TestHDFSFileSystemContract
          org.apache.hadoop.hdfs.server.namenode.TestAddBlock
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
          org.apache.hadoop.security.TestPermissionSymlinks
          org.apache.hadoop.hdfs.server.namenode.TestINodeFile
          org.apache.hadoop.hdfs.TestSetrepDecreasing
          org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
          org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
          org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
          org.apache.hadoop.hdfs.TestListFilesInFileContext
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl
          org.apache.hadoop.hdfs.TestDataTransferProtocol
          org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
          org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations
          org.apache.hadoop.hdfs.server.namenode.TestMetadataVersionOutput
          org.apache.hadoop.hdfs.TestDecommission
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean
          org.apache.hadoop.hdfs.TestSeekBug
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          org.apache.hadoop.hdfs.security.token.block.TestBlockToken
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
          org.apache.hadoop.hdfs.TestDFSFinalize
          org.apache.hadoop.hdfs.TestDFSMkdirs
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename
          org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
          org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
          org.apache.hadoop.hdfs.server.namenode.TestLeaseManager
          org.apache.hadoop.hdfs.server.namenode.TestFSImage
          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
          org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl
          org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
          org.apache.hadoop.hdfs.web.TestWebHdfsTokens
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
          org.apache.hadoop.hdfs.server.namenode.TestSequentialBlockId
          org.apache.hadoop.hdfs.server.namenode.TestMetaSave
          org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
          org.apache.hadoop.hdfs.TestDatanodeRegistration
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
          org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
          org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction
          org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean
          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.TestDatanodeConfig
          org.apache.hadoop.hdfs.server.datanode.TestTriggerBlockReport
          org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          org.apache.hadoop.hdfs.TestFSInputChecker
          org.apache.hadoop.hdfs.server.namenode.TestFileLimit
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
          org.apache.hadoop.hdfs.TestDFSShellGenericOptions
          org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy
          org.apache.hadoop.hdfs.web.TestWebHDFSAcl
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
          org.apache.hadoop.hdfs.TestIsMethodSupported
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
          org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
          org.apache.hadoop.hdfs.TestFileAppendRestart
          org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr
          org.apache.hadoop.hdfs.server.datanode.TestDataStorage
          org.apache.hadoop.hdfs.TestDatanodeReport
          org.apache.hadoop.hdfs.TestMiniDFSCluster
          org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
          org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
          org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
          org.apache.hadoop.hdfs.TestFsShellPermission
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
          org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag
          org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount
          org.apache.hadoop.hdfs.web.TestWebHDFSForHA
          org.apache.hadoop.hdfs.TestEncryptionZonesWithHA
          org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
          org.apache.hadoop.hdfs.server.namenode.TestCreateEditsLog
          org.apache.hadoop.hdfs.server.namenode.TestFSNamesystemMBean
          org.apache.hadoop.hdfs.TestFileStatus
          org.apache.hadoop.hdfs.web.TestWebHDFSXAttr
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr
          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          org.apache.hadoop.hdfs.server.datanode.TestHSync
          org.apache.hadoop.hdfs.server.datanode.TestStorageReport
          org.apache.hadoop.hdfs.TestBalancerBandwidth
          org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          org.apache.hadoop.hdfs.TestListFilesInDFS
          org.apache.hadoop.hdfs.server.namenode.TestHostsFiles
          org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          org.apache.hadoop.hdfs.TestDFSRename
          org.apache.hadoop.security.TestRefreshUserMappings
          org.apache.hadoop.hdfs.security.TestDelegationToken
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean
          org.apache.hadoop.security.TestPermission
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
          org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
          org.apache.hadoop.hdfs.web.TestWebHDFS
          org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8595//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8595//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8595//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678104/HDFS-7035.014.patch against trunk revision 2a6be65. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 javac . The applied patch generated 1266 javac compiler warnings (more than the trunk's current 1265 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestAllowFormat org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestEncryptedTransfer org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream org.apache.hadoop.hdfs.TestSnapshotCommands org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing org.apache.hadoop.hdfs.TestRead org.apache.hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots org.apache.hadoop.hdfs.TestBlocksScheduledCounter org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate org.apache.hadoop.hdfs.TestDFSPermission org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.server.namenode.TestStartup org.apache.hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics org.apache.hadoop.hdfs.server.namenode.TestFSImageWithXAttr org.apache.hadoop.hdfs.TestDFSClientFailover org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl org.apache.hadoop.hdfs.TestWriteConfigurationToDFS org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation org.apache.hadoop.hdfs.web.TestHttpsFileSystem org.apache.hadoop.hdfs.server.namenode.TestFSDirectory org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles org.apache.hadoop.hdfs.TestDFSOutputStream org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode org.apache.hadoop.hdfs.TestDFSRollback org.apache.hadoop.hdfs.TestClientBlockVerification org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN org.apache.hadoop.hdfs.TestFileCreationEmpty org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.server.datanode.TestDiskError org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.server.namenode.TestNameNodeRpcServer org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes org.apache.hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache org.apache.hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.TestReadWhileWriting org.apache.hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.server.namenode.TestAddBlock org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics org.apache.hadoop.security.TestPermissionSymlinks org.apache.hadoop.hdfs.server.namenode.TestINodeFile org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations org.apache.hadoop.hdfs.server.namenode.TestMetadataVersionOutput org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean org.apache.hadoop.hdfs.TestSeekBug org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.security.token.block.TestBlockToken org.apache.hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.TestDFSMkdirs org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename org.apache.hadoop.hdfs.TestDFSClientExcludedNodes org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.server.namenode.TestLeaseManager org.apache.hadoop.hdfs.server.namenode.TestFSImage org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength org.apache.hadoop.hdfs.web.TestWebHdfsTokens org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport org.apache.hadoop.hdfs.server.namenode.TestSequentialBlockId org.apache.hadoop.hdfs.server.namenode.TestMetaSave org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner org.apache.hadoop.hdfs.TestDatanodeRegistration org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestDatanodeConfig org.apache.hadoop.hdfs.server.datanode.TestTriggerBlockReport org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation org.apache.hadoop.hdfs.TestFSInputChecker org.apache.hadoop.hdfs.server.namenode.TestFileLimit org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing org.apache.hadoop.hdfs.TestDFSShellGenericOptions org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy org.apache.hadoop.hdfs.web.TestWebHDFSAcl org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles org.apache.hadoop.hdfs.TestIsMethodSupported org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot org.apache.hadoop.hdfs.server.namenode.TestEditLogRace org.apache.hadoop.hdfs.TestFileAppendRestart org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr org.apache.hadoop.hdfs.server.datanode.TestDataStorage org.apache.hadoop.hdfs.TestDatanodeReport org.apache.hadoop.hdfs.TestMiniDFSCluster org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus org.apache.hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot org.apache.hadoop.hdfs.TestFsShellPermission org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.hdfs.web.TestWebHDFSForHA org.apache.hadoop.hdfs.TestEncryptionZonesWithHA org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives org.apache.hadoop.hdfs.server.namenode.TestCreateEditsLog org.apache.hadoop.hdfs.server.namenode.TestFSNamesystemMBean org.apache.hadoop.hdfs.TestFileStatus org.apache.hadoop.hdfs.web.TestWebHDFSXAttr org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestHSync org.apache.hadoop.hdfs.server.datanode.TestStorageReport org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork org.apache.hadoop.hdfs.TestListFilesInDFS org.apache.hadoop.hdfs.server.namenode.TestHostsFiles org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot org.apache.hadoop.hdfs.TestDFSRename org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotStatsMXBean org.apache.hadoop.security.TestPermission org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner org.apache.hadoop.hdfs.web.TestWebHDFS org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8595//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8595//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8595//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Fixed a typo that causes wrong condition checking in BlockPoolSliceStorage#loadStorageDirectory.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Fixed a typo that causes wrong condition checking in BlockPoolSliceStorage#loadStorageDirectory .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678252/HDFS-7035.015.patch
          against trunk revision b811212.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 javac. The applied patch generated 1266 javac compiler warnings (more than the trunk's current 1265 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8599//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8599//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8599//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678252/HDFS-7035.015.patch against trunk revision b811212. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 javac . The applied patch generated 1266 javac compiler warnings (more than the trunk's current 1265 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8599//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8599//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8599//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Updated the patch to fix javac warnings.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Updated the patch to fix javac warnings.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678313/HDFS-7035.016.patch
          against trunk revision 5e3f428.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8602//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678313/HDFS-7035.016.patch against trunk revision 5e3f428. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8602//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Fix conflicts against trunk.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Fix conflicts against trunk.
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1 pending jenkins. Thanks, Eddy.

          Show
          cmccabe Colin P. McCabe added a comment - +1 pending jenkins. Thanks, Eddy.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678321/HDFS-7035.016.patch
          against trunk revision 5e3f428.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestBlockReaderLocal

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8604//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8604//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678321/HDFS-7035.016.patch against trunk revision 5e3f428. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestBlockReaderLocal +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8604//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8604//console This message is automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Thanks for your +1, Colin P. McCabe.

          The failure is a process killed by JVM, which seems not related to this patch.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Thanks for your +1, Colin P. McCabe . The failure is a process killed by JVM, which seems not related to this patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678321/HDFS-7035.016.patch
          against trunk revision 81fe8e4.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestEncryptedTransfer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8606//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8606//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678321/HDFS-7035.016.patch against trunk revision 81fe8e4. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptedTransfer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8606//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8606//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Committed to trunk and 2.7. Thanks, Lei (Eddy) Xu.

          Show
          cmccabe Colin P. McCabe added a comment - Committed to trunk and 2.7. Thanks, Lei (Eddy) Xu .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #6401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6401/)
          HDFS-7035. Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #6401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6401/ ) HDFS-7035 . Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #729 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/729/)
          HDFS-7035. Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #729 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/729/ ) HDFS-7035 . Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1918 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1918/)
          HDFS-7035. Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1918 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1918/ ) HDFS-7035 . Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1943 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1943/)
          HDFS-7035. Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1943 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1943/ ) HDFS-7035 . Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) (cmccabe: rev a9331fe9b071fdcdae0c6c747d7b6b306142e671) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Sangjin Lee backported this to 2.6.1. I just pushed the commit after running compilation and TestDataNodeHotSwapVolumes, TestDataStorage and TestFsDatasetImpl which changed in the patch.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Sangjin Lee backported this to 2.6.1. I just pushed the commit after running compilation and TestDataNodeHotSwapVolumes, TestDataStorage and TestFsDatasetImpl which changed in the patch.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Lei (Eddy) Xu, one question related to this change. DataNode#refreshVolumes sets DFS_DATANODE_DATA_DIR_KEY to effectiveVolumes. This can be different from the input list passed in as a parameter as it filters out failed volumes.

                conf.set(DFS_DATANODE_DATA_DIR_KEY,
                    Joiner.on(",").join(effectiveVolumes));
                dataDirs = getStorageLocations(conf);
          

          However this will conflict with the value set by ReconfigurableBase#reconfigureProperty when DataNode#reconfigurePropertyImpl returns:

                  reconfigurePropertyImpl(property, newVal);
                  if (newVal != null) {
                    getConf().set(property, newVal);
                  } else {
                    getConf().unset(property);
                  }
          

          Does this need fixing or am I understanding it wrong? Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Lei (Eddy) Xu , one question related to this change. DataNode#refreshVolumes sets DFS_DATANODE_DATA_DIR_KEY to effectiveVolumes. This can be different from the input list passed in as a parameter as it filters out failed volumes. conf.set(DFS_DATANODE_DATA_DIR_KEY, Joiner.on( "," ).join(effectiveVolumes)); dataDirs = getStorageLocations(conf); However this will conflict with the value set by ReconfigurableBase#reconfigureProperty when DataNode#reconfigurePropertyImpl returns: reconfigurePropertyImpl(property, newVal); if (newVal != null ) { getConf().set(property, newVal); } else { getConf().unset(property); } Does this need fixing or am I understanding it wrong? Thanks.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Arpit Agarwal

          Currently, we do not enable ReconfigureServlet to call reconfigureProperty, in DN.

          The reason that it only stores effective volumes, is if there is an disk error in the how swap process, we would like the admin to replace the disks and directly run reconfig again.

          If it is a concern in HDFS-9330, can we suggest reconfigurePropertyImpl() to return a value that is stored in conf eventually?

          Thanks.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Arpit Agarwal Currently, we do not enable ReconfigureServlet to call reconfigureProperty , in DN. The reason that it only stores effective volumes, is if there is an disk error in the how swap process, we would like the admin to replace the disks and directly run reconfig again. If it is a concern in HDFS-9330 , can we suggest reconfigurePropertyImpl() to return a value that is stored in conf eventually? Thanks.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          If it is a concern in HDFS-9330, can we suggest reconfigurePropertyImpl() to return a value that is stored in conf eventually?

          Thanks Lei (Eddy) Xu, this is a good suggestion. Not sure we can change the behavior of reconfigurePropertyImpl for backwards-compatibility. We never published ReconfigurableBase as a public API so it may be okay.

          If we want to be conservative and retain compatibility we can add a new overload with a default implementation that throws. ReconfigurableBase would have to try both. I'll file a jira.

          Show
          arpitagarwal Arpit Agarwal added a comment - If it is a concern in HDFS-9330 , can we suggest reconfigurePropertyImpl() to return a value that is stored in conf eventually? Thanks Lei (Eddy) Xu , this is a good suggestion. Not sure we can change the behavior of reconfigurePropertyImpl for backwards-compatibility. We never published ReconfigurableBase as a public API so it may be okay. If we want to be conservative and retain compatibility we can add a new overload with a default implementation that throws. ReconfigurableBase would have to try both. I'll file a jira.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I filed HADOOP-12746.

          Show
          arpitagarwal Arpit Agarwal added a comment - I filed HADOOP-12746 .
          Hide
          kanaka Kanaka Kumar Avvaru added a comment -

          The following code in the patch is a dead code now as "initialized" is false and never set to true.

          +    if (this.initialized) {
          +      LOG.info("DataNode version: " + HdfsConstants.DATANODE_LAYOUT_VERSION
          +          + " and NameNode layout version: " + nsInfo.getLayoutVersion());
          +      this.storageDirs = new ArrayList<StorageDirectory>(dataDirs.size());
          +      // mark DN storage is initialized
          +      this.initialized = true;
          

          IIUC , recoverTransitionRead is called only once after DN started during block pool initialization. So can we remove initialized flag & this unused code ?

          Show
          kanaka Kanaka Kumar Avvaru added a comment - The following code in the patch is a dead code now as "initialized" is false and never set to true. + if ( this .initialized) { + LOG.info( "DataNode version: " + HdfsConstants.DATANODE_LAYOUT_VERSION + + " and NameNode layout version: " + nsInfo.getLayoutVersion()); + this .storageDirs = new ArrayList<StorageDirectory>(dataDirs.size()); + // mark DN storage is initialized + this .initialized = true ; IIUC , recoverTransitionRead is called only once after DN started during block pool initialization. So can we remove initialized flag & this unused code ?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Kanaka Kumar Avvaru thanks for reporting this.

          HDFS-9372 address the same..we can backport branch-2.7 also.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Kanaka Kumar Avvaru thanks for reporting this. HDFS-9372 address the same..we can backport branch-2.7 also.

            People

            • Assignee:
              eddyxu Lei (Eddy) Xu
              Reporter:
              eddyxu Lei (Eddy) Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development