Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: balancer & mover, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In most of the Hadoop clusters, as more and more data is stored for longer time, the demand for storage is outstripping the compute. Hadoop needs a cost effective and easy to manage solution to meet this demand for storage. Current solution is:

      • Delete the old unused data. This comes at operational cost of identifying unnecessary data and deleting them manually.
      • Add more nodes to the clusters. This adds along with storage capacity unnecessary compute capacity to the cluster.

      Hadoop needs a solution to decouple growing storage capacity from compute capacity. Nodes with higher density and less expensive storage with low compute power are becoming available and can be used as cold storage in the clusters. Based on policy the data from hot storage can be moved to cold storage. Adding more nodes to the cold storage can grow the storage independent of the compute capacity in the cluster.

      1. HDFSArchivalStorageDesign20140715.pdf
        166 kB
        Tsz Wo Nicholas Sze
      2. HDFSArchivalStorageDesign20140623.pdf
        152 kB
        Tsz Wo Nicholas Sze
      3. HDFS-6584.000.patch
        344 kB
        Jing Zhao
      4. h6584_20140918b.patch
        423 kB
        Tsz Wo Nicholas Sze
      5. h6584_20140918.patch
        425 kB
        Tsz Wo Nicholas Sze
      6. h6584_20140917b.patch
        420 kB
        Jing Zhao
      7. h6584_20140917.patch
        420 kB
        Jing Zhao
      8. h6584_20140916.patch
        402 kB
        Jing Zhao
      9. h6584_20140916.patch
        400 kB
        Jing Zhao
      10. h6584_20140915.patch
        397 kB
        Jing Zhao
      11. h6584_20140911b.patch
        394 kB
        Jing Zhao
      12. h6584_20140911.patch
        395 kB
        Tsz Wo Nicholas Sze
      13. h6584_20140908b.patch
        388 kB
        Jing Zhao
      14. h6584_20140908.patch
        373 kB
        Tsz Wo Nicholas Sze
      15. h6584_20140907.patch
        372 kB
        Tsz Wo Nicholas Sze
      16. archival-storage-testplan.pdf
        45 kB
        Jing Zhao

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          marcuspaget Mark Paget added a comment -

          Perhaps rack topology could allow for tagging low priority. Then manual mechanism to tag or automation for least frequently used.

          Show
          marcuspaget Mark Paget added a comment - Perhaps rack topology could allow for tagging low priority. Then manual mechanism to tag or automation for least frequently used.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          HDFSArchivalStorageDesign20140623.pdf: design doc.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - HDFSArchivalStorageDesign20140623.pdf: design doc.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I just have created a branch http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-6584/ and added a fixed version for the Archival Storage development.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I just have created a branch http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-6584/ and added a fixed version for the Archival Storage development.
          Hide
          nroberts Nathan Roberts added a comment -

          As part of this work, would it make sense to extend the policy functionality so that we can control the BlockPlacementPolicy used during file creation? Here's the use case which we run into quite frequently: 1-2GB file generated by JobA is used as a distributed cache file for Job B which is rather large (several thousand tasks). Assuming a single task from JobA writes this file, an entire copy of the file will be on a single node, and in no other nodes in that same rack (default policy is 1st blk local, 2nd replica on remote rack, 3rd replica on same rack as 2nd). When a large job needs this distributed cache file, the node where there is a single copy will become a significant bottleneck and is likely to cause localization timeouts. This is with replication factors set to 50+, so just increasing the replication factor does not solve this problem. It would be good if JobA could specify a BlockPlacementPolicy which would do 1st replica rack local, 2nd replica remote rack, 3rd replica same as 2nd (in general though it would be good if JobA could ask for any 1 of n placement policies).

          Show
          nroberts Nathan Roberts added a comment - As part of this work, would it make sense to extend the policy functionality so that we can control the BlockPlacementPolicy used during file creation? Here's the use case which we run into quite frequently: 1-2GB file generated by JobA is used as a distributed cache file for Job B which is rather large (several thousand tasks). Assuming a single task from JobA writes this file, an entire copy of the file will be on a single node, and in no other nodes in that same rack (default policy is 1st blk local, 2nd replica on remote rack, 3rd replica on same rack as 2nd). When a large job needs this distributed cache file, the node where there is a single copy will become a significant bottleneck and is likely to cause localization timeouts. This is with replication factors set to 50+, so just increasing the replication factor does not solve this problem. It would be good if JobA could specify a BlockPlacementPolicy which would do 1st replica rack local, 2nd replica remote rack, 3rd replica same as 2nd (in general though it would be good if JobA could ask for any 1 of n placement policies).
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Hi Nathan, we already support excludedNodes in NameNodeRpcServer.addBlock(..). So, client could simply put localhost in excludedNodes. Then, we will get 1st replica rack local, 2nd replica remote rack, 3rd replica same as 2nd. However, we still need to change the client API for putting localhost in excludedNodes.

          I think this is a small simple change. Let's just create a JIRA for it?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Hi Nathan, we already support excludedNodes in NameNodeRpcServer.addBlock(..). So, client could simply put localhost in excludedNodes. Then, we will get 1st replica rack local, 2nd replica remote rack, 3rd replica same as 2nd. However, we still need to change the client API for putting localhost in excludedNodes. I think this is a small simple change. Let's just create a JIRA for it?
          Hide
          shv Konstantin Shvachko added a comment -

          The design doc talks about the default replication of 3. What happens with other replicas, if I set file replication to 10 or 4095? With Hot and Cold one can assume All DISK
          and All ARCHIVE, resp. But with Warm there are options for higher replication factors.

          Show
          shv Konstantin Shvachko added a comment - The design doc talks about the default replication of 3. What happens with other replicas, if I set file replication to 10 or 4095? With Hot and Cold one can assume All DISK and All ARCHIVE, resp. But with Warm there are options for higher replication factors.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Good question. Here is more details:

              A list of storage types for storing the block replicas such as
          
                STORAGE_TYPE_1, STORAGE_TYPE_2, ..., STORAGE_TYPE_n
            
              When creating a block, the i-th replica is stored using i-th storage type
              for i less than or equal to n, and
              the j-th replica is stored using n-th storage type for j greater than n.
          
              Empty list is not allowed.
          
              Examples:
              DISK          : all replicas stored using DISK.
              DISK, ARCHIVE : the first replica is stored using DISK and all the
                              remaining replicas are stored using ARCHIVE.
          

          The above is copied from the patch in HDFS-6670.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Good question. Here is more details: A list of storage types for storing the block replicas such as STORAGE_TYPE_1, STORAGE_TYPE_2, ..., STORAGE_TYPE_n When creating a block, the i-th replica is stored using i-th storage type for i less than or equal to n, and the j-th replica is stored using n-th storage type for j greater than n. Empty list is not allowed. Examples: DISK : all replicas stored using DISK. DISK, ARCHIVE : the first replica is stored using DISK and all the remaining replicas are stored using ARCHIVE. The above is copied from the patch in HDFS-6670 .
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          HDFSArchivalStorageDesign20140715.pdf: revised design doc.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - HDFSArchivalStorageDesign20140715.pdf: revised design doc.
          Hide
          jingzhao Jing Zhao added a comment - - edited

          Upload a consolidated patch to run Jenkins.

          Show
          jingzhao Jing Zhao added a comment - - edited Upload a consolidated patch to run Jenkins.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h6997_20140907.patch: synced with new commits.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h6997_20140907.patch: synced with new commits.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch
          against trunk revision d1fa582.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7935//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch against trunk revision d1fa582. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7935//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch
          against trunk revision d1fa582.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7936//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch against trunk revision d1fa582. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7936//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch
          against trunk revision d1fa582.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7937//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch against trunk revision d1fa582. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7937//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch
          against trunk revision d1fa582.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7938//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667081/h6997_20140907.patch against trunk revision d1fa582. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7938//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Oops, uploaded a wrong file. The file should be h6584_20140907.patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Oops, uploaded a wrong file. The file should be h6584_20140907.patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667084/h6584_20140907.patch
          against trunk revision d1fa582.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified test files.

          -1 javac. The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1264 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          org.apache.hadoop.hdfs.server.namenode.TestINodeFile
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.hdfs.TestEncryptionZones
          org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl
          org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.fs.TestSymlinkHdfsFileContext
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          org.apache.hadoop.hdfs.TestListFilesInFileContext
          org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream
          org.apache.hadoop.hdfs.server.namenode.TestFSImage
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7939//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7939//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7939//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667084/h6584_20140907.patch against trunk revision d1fa582. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 23 new or modified test files. -1 javac . The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1264 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes org.apache.hadoop.hdfs.server.namenode.TestINodeFile org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.TestEncryptionZones org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.fs.TestSymlinkHdfsFileContext org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.fs.TestSymlinkHdfsFileSystem org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream org.apache.hadoop.hdfs.server.namenode.TestFSImage org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7939//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7939//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7939//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h6584_20140908.patch: with HDFS-7028.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h6584_20140908.patch: with HDFS-7028 .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667122/h6584_20140908.patch
          against trunk revision 0974f43.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified test files.

          -1 javac. The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1264 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7945//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7945//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7945//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667122/h6584_20140908.patch against trunk revision 0974f43. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 23 new or modified test files. -1 javac . The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1264 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7945//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7945//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7945//console This message is automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Post the test plan originally written by Yesha Vora.

          Show
          jingzhao Jing Zhao added a comment - Post the test plan originally written by Yesha Vora .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667240/archival-storage-testplan.pdf
          against trunk revision d989ac0.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7955//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667240/archival-storage-testplan.pdf against trunk revision d989ac0. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7955//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667278/h6584_20140908b.patch
          against trunk revision 7498dd7.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7960//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667278/h6584_20140908b.patch against trunk revision 7498dd7. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7960//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h7039_20140911.patch: with HDFS-7034 and HDFS-7039.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h7039_20140911.patch: with HDFS-7034 and HDFS-7039 .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667961/h7039_20140911.patch
          against trunk revision 4be9517.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7994//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667961/h7039_20140911.patch against trunk revision 4be9517. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7994//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Oops, uploaded a wrong file. Here is the correct one.

          h6584_20140911.patch

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Oops, uploaded a wrong file. Here is the correct one. h6584_20140911.patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12667981/h6584_20140911.patch
          against trunk revision 4be9517.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          -1 javac. The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1264 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestEncryptionZones
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.TestPersistBlocks

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7996//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7996//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7996//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667981/h6584_20140911.patch against trunk revision 4be9517. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. -1 javac . The applied patch generated 1268 javac compiler warnings (more than the trunk's current 1264 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptionZones org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.hdfs.server.datanode.TestBPOfferService org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.TestPersistBlocks The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7996//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7996//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7996//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668173/h6584_20140911b.patch
          against trunk revision c656d7d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8000//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8000//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668173/h6584_20140911b.patch against trunk revision c656d7d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8000//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8000//console This message is automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi all,

          Nice work here! A few of us here at Cloudera went through the design doc and some of the code, and had a few questions. I don't think any of this is a blocker for the merge, but I'd like to see most of these answered in the design doc as well as the user documentation (which I see hasn't been completed yet). The "data migration tool" section especially could use some elaboration.

          New Mover daemon

          • When does the Mover actually migrate data? When a block is finalized? When the file is closed? Some amount of time after? When the admin decides to run the Mover?
          • Since the Mover is based on the Balancer, is there any concern about it being too slow to move data from fast storage to archival? If all data migrates off to archival, the mover needs to keep up with the aggregate write rate of the cluster. The balancer, putting it mildly, is not the fastest tool in this regard.
          • How does this interact with open files?
          • What is the load impact of scanning the namespace for files that need to be migrated? A naive ls -R / type operation could be bad.

          Storage policies

          • Why are policies specified in XML files rather than in the fsimage / edit log? It seems very important to keep the policies consistent, and this is thus one more file that needs to be synchronized and backed up. Stashing it in the editlog would do this for you.
          • Related, if the XML file disappears or isn't synchronized properly, isn't there potential for big data movements? For instance, I could easily see an admin accidentally running the Mover from a node without the XML file and then possibly bad things happening.
          • The previous might motivate an RPC interface for getting the storage policies from the NN.
          • Can storage policies be set at a directory level? Testing to confirm this either way?
          • How does this interact with snapshots? With replication factor, I believe we use the maximum replication factor across all snapshots. Here, would it be the union of all storage types across all snapshots? Not sure how the Mover accounts for this, or if a full-union is the right policy.
          • Do we have per-storage-type quotas? Are there APIs exposed to show, for instance, storage type usage by a snapshot, by a directory, etc?

          API

          • We exposed cachedHosts in BlockLocation, so application schedulers can choose to place their tasks for cache locality. We need a similar thing for storage type, so schedulers can prefer "hotter" replicas.

          Thanks guys!

          Show
          andrew.wang Andrew Wang added a comment - Hi all, Nice work here! A few of us here at Cloudera went through the design doc and some of the code, and had a few questions. I don't think any of this is a blocker for the merge, but I'd like to see most of these answered in the design doc as well as the user documentation (which I see hasn't been completed yet). The "data migration tool" section especially could use some elaboration. New Mover daemon When does the Mover actually migrate data? When a block is finalized? When the file is closed? Some amount of time after? When the admin decides to run the Mover? Since the Mover is based on the Balancer, is there any concern about it being too slow to move data from fast storage to archival? If all data migrates off to archival, the mover needs to keep up with the aggregate write rate of the cluster. The balancer, putting it mildly, is not the fastest tool in this regard. How does this interact with open files? What is the load impact of scanning the namespace for files that need to be migrated? A naive ls -R / type operation could be bad. Storage policies Why are policies specified in XML files rather than in the fsimage / edit log? It seems very important to keep the policies consistent, and this is thus one more file that needs to be synchronized and backed up. Stashing it in the editlog would do this for you. Related, if the XML file disappears or isn't synchronized properly, isn't there potential for big data movements? For instance, I could easily see an admin accidentally running the Mover from a node without the XML file and then possibly bad things happening. The previous might motivate an RPC interface for getting the storage policies from the NN. Can storage policies be set at a directory level? Testing to confirm this either way? How does this interact with snapshots? With replication factor, I believe we use the maximum replication factor across all snapshots. Here, would it be the union of all storage types across all snapshots? Not sure how the Mover accounts for this, or if a full-union is the right policy. Do we have per-storage-type quotas? Are there APIs exposed to show, for instance, storage type usage by a snapshot, by a directory, etc? API We exposed cachedHosts in BlockLocation, so application schedulers can choose to place their tasks for cache locality. We need a similar thing for storage type, so schedulers can prefer "hotter" replicas. Thanks guys!
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks a lot for the great comments, Andrew Wang! Let me try to answer some of the questions here, and I believe Tsz Wo Nicholas Sze will provide more details later.

          When does the Mover actually migrate data? When a block is finalized? When the file is closed? Some amount of time after? When the admin decides to run the Mover?

          Currently the data is only migrated when admin runs the Mover.

          What is the load impact of scanning the namespace for files that need to be migrated? A naive ls -R / type operation could be bad.

          Yeah, scanning the namespace is definitely a big burden here. HDFS-6875 adds the support to allow users to specify a list of paths for migration. And in the future we may want to support running multiple Movers for disjoint directories concurrently or even utilizing MR.

          Why are policies specified in XML files rather than in the fsimage / edit log? It seems very important to keep the policies consistent, and this is thus one more file that needs to be synchronized and backed up. Stashing it in the editlog would do this for you.

          Agree. Actually Nicholas and I had a discussion about this before, and I had a unfinished preliminary patch but still need to think more about some details. We plan to finish this work after the merge.

          Can storage policies be set at a directory level? Testing to confirm this either way?

          Yes, this has been done in HDFS-6847.

          How does this interact with snapshots? With replication factor, I believe we use the maximum replication factor across all snapshots. Here, would it be the union of all storage types across all snapshots? Not sure how the Mover accounts for this, or if a full-union is the right policy.

          This has been addressed in HDFS-6969. Please see the discussion there.

          Do we have per-storage-type quotas? Are there APIs exposed to show, for instance, storage type usage by a snapshot, by a directory, etc?

          This is a very good suggestion, especially considering we also have storage type SSD and in the future we may also have storage type MEMORY.

          How does this interact with open files?

          Actually we should ignore the incomplete block which can be inferred from LocatedBlocks. I will file a new jira for this. Thanks!
          In another scenario, if a block later gets appended during the migration, the new replica will be marked as corrupted when it is reported to the NN because of the inconsistency of generation stamp.

          Show
          jingzhao Jing Zhao added a comment - Thanks a lot for the great comments, Andrew Wang ! Let me try to answer some of the questions here, and I believe Tsz Wo Nicholas Sze will provide more details later. When does the Mover actually migrate data? When a block is finalized? When the file is closed? Some amount of time after? When the admin decides to run the Mover? Currently the data is only migrated when admin runs the Mover. What is the load impact of scanning the namespace for files that need to be migrated? A naive ls -R / type operation could be bad. Yeah, scanning the namespace is definitely a big burden here. HDFS-6875 adds the support to allow users to specify a list of paths for migration. And in the future we may want to support running multiple Movers for disjoint directories concurrently or even utilizing MR. Why are policies specified in XML files rather than in the fsimage / edit log? It seems very important to keep the policies consistent, and this is thus one more file that needs to be synchronized and backed up. Stashing it in the editlog would do this for you. Agree. Actually Nicholas and I had a discussion about this before, and I had a unfinished preliminary patch but still need to think more about some details. We plan to finish this work after the merge. Can storage policies be set at a directory level? Testing to confirm this either way? Yes, this has been done in HDFS-6847 . How does this interact with snapshots? With replication factor, I believe we use the maximum replication factor across all snapshots. Here, would it be the union of all storage types across all snapshots? Not sure how the Mover accounts for this, or if a full-union is the right policy. This has been addressed in HDFS-6969 . Please see the discussion there. Do we have per-storage-type quotas? Are there APIs exposed to show, for instance, storage type usage by a snapshot, by a directory, etc? This is a very good suggestion, especially considering we also have storage type SSD and in the future we may also have storage type MEMORY. How does this interact with open files? Actually we should ignore the incomplete block which can be inferred from LocatedBlocks. I will file a new jira for this. Thanks! In another scenario, if a block later gets appended during the migration, the new replica will be marked as corrupted when it is reported to the NN because of the inconsistency of generation stamp.
          Hide
          jingzhao Jing Zhao added a comment -

          How does this interact with open files?

          Actually we should ignore the incomplete block which can be inferred from LocatedBlocks.

          Checked the code. Looks like the current writing pipeline handles this scenario correctly (without causing data corruption). The namenode will not delete any replica when it finds that the block is actually still under construction. But it will still be more efficient to avoid migrating under-construction blocks.

          Show
          jingzhao Jing Zhao added a comment - How does this interact with open files? Actually we should ignore the incomplete block which can be inferred from LocatedBlocks. Checked the code. Looks like the current writing pipeline handles this scenario correctly (without causing data corruption). The namenode will not delete any replica when it finds that the block is actually still under construction. But it will still be more efficient to avoid migrating under-construction blocks.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > ... A few of us here at Cloudera went through the design doc and some of the code, and had a few questions. ...

          Andrew Wang and others, thanks for reviewing the design doc and the code. Would you mind listing out the other people at Cloudera who have participated the review (unless they want to stay anonymous) so that we could possibly acknowledge them in the future?

          > ... Let me try to answer some of the questions here, and I believe Tsz Wo Nicholas Sze will provide more details later.

          Thanks Jing Zhao for answering the questions. You actually have provided a lot of details.

          Andrew Wang, if you have further questions, please feel free to let us know.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > ... A few of us here at Cloudera went through the design doc and some of the code, and had a few questions. ... Andrew Wang and others, thanks for reviewing the design doc and the code. Would you mind listing out the other people at Cloudera who have participated the review (unless they want to stay anonymous) so that we could possibly acknowledge them in the future? > ... Let me try to answer some of the questions here, and I believe Tsz Wo Nicholas Sze will provide more details later. Thanks Jing Zhao for answering the questions. You actually have provided a lot of details. Andrew Wang , if you have further questions, please feel free to let us know.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the responses Jing, very helpful. For the watchers at home, some quick tl;dr summaries are that the latest storage policy is used for a snapshotted file, and storage policies on nested directories use the closest ancestor policy for a file.

          Hi Nicholas, basically all of us got in a room and discussed the design doc, meaning myself, Colin, ATM, Eddy, Zhe, and Charles (hope I didn't miss anyone). I was taking the meeting notes, so I posted the JIRA comment.

          There are still a few questions that Jing left unanswered, pasted below:

          • Since the Mover is based on the Balancer, is there any concern about it being too slow to move data from fast storage to archival? If all data migrates off to archival, the mover needs to keep up with the aggregate write rate of the cluster. The balancer, putting it mildly, is not the fastest tool in this regard.
          • We exposed cachedHosts in BlockLocation, so application schedulers can choose to place their tasks for cache locality. We need a similar thing for storage type, so schedulers can prefer "hotter" replicas.

          I also want to reiterate my request for a design doc / user doc rev. I think basically all the answers here should be addressed in one or the other. I especially would like to know more about the API changes that are planned post-merge. Jing mentioned getting rid of the XML config and putting it in the edit log. This is fairly substantial (new RPCs / cmds), and I expect this would be done before merging things to branch-2 as it has compat implications. Exposing the storage type is another new API, though more minor.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the responses Jing, very helpful. For the watchers at home, some quick tl;dr summaries are that the latest storage policy is used for a snapshotted file, and storage policies on nested directories use the closest ancestor policy for a file. Hi Nicholas, basically all of us got in a room and discussed the design doc, meaning myself, Colin, ATM, Eddy, Zhe, and Charles (hope I didn't miss anyone). I was taking the meeting notes, so I posted the JIRA comment. There are still a few questions that Jing left unanswered, pasted below: Since the Mover is based on the Balancer, is there any concern about it being too slow to move data from fast storage to archival? If all data migrates off to archival, the mover needs to keep up with the aggregate write rate of the cluster. The balancer, putting it mildly, is not the fastest tool in this regard. We exposed cachedHosts in BlockLocation, so application schedulers can choose to place their tasks for cache locality. We need a similar thing for storage type, so schedulers can prefer "hotter" replicas. I also want to reiterate my request for a design doc / user doc rev. I think basically all the answers here should be addressed in one or the other. I especially would like to know more about the API changes that are planned post-merge. Jing mentioned getting rid of the XML config and putting it in the edit log. This is fairly substantial (new RPCs / cmds), and I expect this would be done before merging things to branch-2 as it has compat implications. Exposing the storage type is another new API, though more minor.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks Andrew Wang!

          Since the Mover is based on the Balancer, is there any concern about it being too slow to move data from fast storage to archival? If all data migrates off to archival, the mover needs to keep up with the aggregate write rate of the cluster. The balancer, putting it mildly, is not the fastest tool in this regard.

          Here are some of my thoughts. Please let me know if I miss something, Tsz Wo Nicholas Sze.
          1) Currently the migration tool still depends on admin to mark files/dirs as COLD/WARM, it may be rare that users still actively writing new data into a
          directory after marking it as COLD. Thus for now this may not be a critical concern.
          2) Tools/services may later be developed to actively/automatically scan the namespace and mark COLD files based on different rules such as access/modification time. In some cases, if the rule is very aggressive and the migration is very slow, we may have the issue you mentioned. The current Mover is utilizing the Dispatcher, or more generally, the DataTransferProtocol#replaceBlock protocol. I guess with more aggressive settings (e.g., the max number of blocks scheduled on each DataNode for migration), the migration speed should not be very slow, and it should be easy for us to replace the Dispatcher with a faster migration framework.

          We exposed cachedHosts in BlockLocation, so application schedulers can choose to place their tasks for cache locality. We need a similar thing for storage type, so schedulers can prefer "hotter" replicas.

          This is a very good suggestion, we can add this information later. Thanks!

          BTW, HDFS-7062 has been committed to fix the open file issue. A doc patch has been uploaded in HDFS-6864. Thanks again for the great comments, Andrew Wang!

          Show
          jingzhao Jing Zhao added a comment - Thanks Andrew Wang ! Since the Mover is based on the Balancer, is there any concern about it being too slow to move data from fast storage to archival? If all data migrates off to archival, the mover needs to keep up with the aggregate write rate of the cluster. The balancer, putting it mildly, is not the fastest tool in this regard. Here are some of my thoughts. Please let me know if I miss something, Tsz Wo Nicholas Sze . 1) Currently the migration tool still depends on admin to mark files/dirs as COLD/WARM, it may be rare that users still actively writing new data into a directory after marking it as COLD. Thus for now this may not be a critical concern. 2) Tools/services may later be developed to actively/automatically scan the namespace and mark COLD files based on different rules such as access/modification time. In some cases, if the rule is very aggressive and the migration is very slow, we may have the issue you mentioned. The current Mover is utilizing the Dispatcher, or more generally, the DataTransferProtocol#replaceBlock protocol. I guess with more aggressive settings (e.g., the max number of blocks scheduled on each DataNode for migration), the migration speed should not be very slow, and it should be easy for us to replace the Dispatcher with a faster migration framework. We exposed cachedHosts in BlockLocation, so application schedulers can choose to place their tasks for cache locality. We need a similar thing for storage type, so schedulers can prefer "hotter" replicas. This is a very good suggestion, we can add this information later. Thanks! BTW, HDFS-7062 has been committed to fix the open file issue. A doc patch has been uploaded in HDFS-6864 . Thanks again for the great comments, Andrew Wang !
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12668803/h6584_20140915.patch
          against trunk revision 43b0303.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8031//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8031//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668803/h6584_20140915.patch against trunk revision 43b0303. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8031//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8031//console This message is automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Upload a patch including HDFS-7072.

          Show
          jingzhao Jing Zhao added a comment - Upload a patch including HDFS-7072 .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669298/h6584_20140916.patch
          against trunk revision 0e7d1db.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8045//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669298/h6584_20140916.patch against trunk revision 0e7d1db. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8045//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669300/h6584_20140916.patch
          against trunk revision 0e7d1db.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.crypto.random.TestOsSecureRandom
          org.apache.hadoop.ha.TestZKFailoverControllerStress
          org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.cli.TestAclCLI
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8046//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8046//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669300/h6584_20140916.patch against trunk revision 0e7d1db. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.crypto.random.TestOsSecureRandom org.apache.hadoop.ha.TestZKFailoverControllerStress org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.cli.TestAclCLI org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8046//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8046//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669300/h6584_20140916.patch
          against trunk revision 0e7d1db.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.crypto.random.TestOsSecureRandom
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8050//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8050//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669300/h6584_20140916.patch against trunk revision 0e7d1db. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.crypto.random.TestOsSecureRandom org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8050//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8050//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669300/h6584_20140916.patch
          against trunk revision 0e7d1db.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.crypto.random.TestOsSecureRandom

          The test build failed in hadoop-hdfs-project/hadoop-hdfs

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8051//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8051//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669300/h6584_20140916.patch against trunk revision 0e7d1db. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.crypto.random.TestOsSecureRandom The test build failed in hadoop-hdfs-project/hadoop-hdfs +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8051//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8051//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669441/h6584_20140917.patch
          against trunk revision c0c7e6f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8054//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8054//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669441/h6584_20140917.patch against trunk revision c0c7e6f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8054//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8054//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669441/h6584_20140917.patch
          against trunk revision ea4e2e8.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8060//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669441/h6584_20140917.patch against trunk revision ea4e2e8. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8060//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669497/h6584_20140917b.patch
          against trunk revision e3803d0.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8061//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669497/h6584_20140917b.patch against trunk revision e3803d0. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8061//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h6584_20140918.patch: with HDFS-7088.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h6584_20140918.patch: with HDFS-7088 .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669669/h6584_20140918.patch
          against trunk revision ee21b13.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8076//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669669/h6584_20140918.patch against trunk revision ee21b13. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8076//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h6584_20140918b.patch: excludes hdfs.cmd since the patch command does not work with dos file correctly.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h6584_20140918b.patch: excludes hdfs.cmd since the patch command does not work with dos file correctly.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669683/h6584_20140918b.patch
          against trunk revision ee21b13.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 28 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ipc.TestFairCallQueue
          org.apache.hadoop.ipc.TestCallQueueManager
          org.apache.hadoop.crypto.random.TestOsSecureRandom
          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.tracing.TestTracing
          org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestHFlush
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8078//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8078//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669683/h6584_20140918b.patch against trunk revision ee21b13. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 28 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ipc.TestFairCallQueue org.apache.hadoop.ipc.TestCallQueueManager org.apache.hadoop.crypto.random.TestOsSecureRandom org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.tracing.TestTracing org.apache.hadoop.hdfs.server.datanode.TestBPOfferService org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8078//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8078//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669683/h6584_20140918b.patch
          against trunk revision ee21b13.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 28 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.mover.TestStorageMover
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8082//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8082//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669683/h6584_20140918b.patch against trunk revision ee21b13. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 28 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.mover.TestStorageMover org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8082//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8082//console This message is automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Failures of TestEncryptionZonesWithKMS, TestWebHdfsFileSystemContract, and TestPipelinesFailover are also seen in other Jenkins run and should be unrelated. Failure of TestOfflineEditsViewer is expected since we need to update the editsStored binary file. Failure of TestStorageMover cannot be reproduced in my local machine (I run the test 100 times but still could not reproduce the failure). Maybe it's related to the Jenkins environment. We can track it in a separate jira.

          I think the feature is ready to be merged into trunk once the vote is closed. Tsz Wo Nicholas Sze, can you close the vote in the dev mailing list?

          Show
          jingzhao Jing Zhao added a comment - Failures of TestEncryptionZonesWithKMS, TestWebHdfsFileSystemContract, and TestPipelinesFailover are also seen in other Jenkins run and should be unrelated. Failure of TestOfflineEditsViewer is expected since we need to update the editsStored binary file. Failure of TestStorageMover cannot be reproduced in my local machine (I run the test 100 times but still could not reproduce the failure). Maybe it's related to the Jenkins environment. We can track it in a separate jira. I think the feature is ready to be merged into trunk once the vote is closed. Tsz Wo Nicholas Sze , can you close the vote in the dev mailing list?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > ... Tsz Wo Nicholas Sze, can you close the vote in the dev mailing list?

          I sent once on Sep 16 but the email somehow did not go out. Just have resent another email.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > ... Tsz Wo Nicholas Sze, can you close the vote in the dev mailing list? I sent once on Sep 16 but the email somehow did not go out. Just have resent another email.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I have merged the branch back to trunk. Thanks everyone who have contributed to this!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I have merged the branch back to trunk. Thanks everyone who have contributed to this!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #685 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/685/)
          Fix hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt for HDFS-6584 after (szetszwo: rev fd3cddf3640a0dbd14556368ae4c6e803083bcfc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Revise the HDFS-6584 entry CHANGES.txt. (szetszwo: rev 5d01a684a38a765eabec53ce88687f1808b6c956)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #685 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/685/ ) Fix hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt for HDFS-6584 after (szetszwo: rev fd3cddf3640a0dbd14556368ae4c6e803083bcfc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Revise the HDFS-6584 entry CHANGES.txt. (szetszwo: rev 5d01a684a38a765eabec53ce88687f1808b6c956) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1901 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1901/)
          Fix hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt for HDFS-6584 after (szetszwo: rev fd3cddf3640a0dbd14556368ae4c6e803083bcfc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Revise the HDFS-6584 entry CHANGES.txt. (szetszwo: rev 5d01a684a38a765eabec53ce88687f1808b6c956)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1901 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1901/ ) Fix hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt for HDFS-6584 after (szetszwo: rev fd3cddf3640a0dbd14556368ae4c6e803083bcfc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Revise the HDFS-6584 entry CHANGES.txt. (szetszwo: rev 5d01a684a38a765eabec53ce88687f1808b6c956) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1876 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/)
          Fix hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt for HDFS-6584 after (szetszwo: rev fd3cddf3640a0dbd14556368ae4c6e803083bcfc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Revise the HDFS-6584 entry CHANGES.txt. (szetszwo: rev 5d01a684a38a765eabec53ce88687f1808b6c956)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1876 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/ ) Fix hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt for HDFS-6584 after (szetszwo: rev fd3cddf3640a0dbd14556368ae4c6e803083bcfc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Revise the HDFS-6584 entry CHANGES.txt. (szetszwo: rev 5d01a684a38a765eabec53ce88687f1808b6c956) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I have merged the HDFS-6584 branch to branch-2.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I have merged the HDFS-6584 branch to branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #6119 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6119/)
          CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #6119 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6119/ ) CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #693 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/693/)
          CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #693 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/693/ ) CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1884 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1884/)
          CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1884 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1884/ ) CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1909 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1909/)
          CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1909 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1909/ ) CHANGES.txt: move HDFS-6584 and its subtasks to Release 2.6.0. (szetszwo: rev a6049aa994dbddc3f640d2121e3aabb3a363943d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Tsz Wo Nicholas Sze,

          Thanks for the work you and other folks did here. I have a question:

          Per your comment:

          https://issues.apache.org/jira/browse/HDFS-6584?focusedCommentId=14139690&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14139690

          https://issues.apache.org/jira/browse/HDFS-6584?focusedCommentId=14148307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14148307

          you merged the feature branch to trunk and branch-2.

          When I look the git log for trunk and branch-2, I can see that trunk has

          commit 22a41dce4af4d5b533ba875b322551db1c152878
          Author: Tsz-Wo Nicholas Sze <szetszwo@hortonworks.com>
          Date:   Sun Sep 7 07:44:28 2014 +0800
          
              HDFS-6997: add more tests for data migration and replicaion.
          

          . However, branch-2 doesn't have it.

          I looked at branch-2, and saw that the HDFS-6997 code is there. I checked another subtask of HDFS-6584, and it's same situation.

          It looks that the commit history is collapsed during the branch-2 merge, but the commit history was kept when doing the trunk merge. Is this intended? would you please comment on what might have happened?

          Thanks much.

          Show
          yzhangal Yongjun Zhang added a comment - HI Tsz Wo Nicholas Sze , Thanks for the work you and other folks did here. I have a question: Per your comment: https://issues.apache.org/jira/browse/HDFS-6584?focusedCommentId=14139690&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14139690 https://issues.apache.org/jira/browse/HDFS-6584?focusedCommentId=14148307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14148307 you merged the feature branch to trunk and branch-2. When I look the git log for trunk and branch-2, I can see that trunk has commit 22a41dce4af4d5b533ba875b322551db1c152878 Author: Tsz-Wo Nicholas Sze <szetszwo@hortonworks.com> Date: Sun Sep 7 07:44:28 2014 +0800 HDFS-6997: add more tests for data migration and replicaion. . However, branch-2 doesn't have it. I looked at branch-2, and saw that the HDFS-6997 code is there. I checked another subtask of HDFS-6584 , and it's same situation. It looks that the commit history is collapsed during the branch-2 merge, but the commit history was kept when doing the trunk merge. Is this intended? would you please comment on what might have happened? Thanks much.

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              szetszwo Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development