Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9255

Consolidate block recovery related implementation into a single class

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. HDFS-9255.01.patch
      43 kB
      Walter Su
    2. HDFS-9255.02.patch
      43 kB
      Walter Su
    3. HDFS-9255.03.patch
      43 kB
      Walter Su
    4. HDFS-9255.04.patch
      44 kB
      Walter Su
    5. HDFS-9255.05.patch
      41 kB
      Walter Su
    6. HDFS-9255.06.patch
      41 kB
      Walter Su
    7. HDFS-9255-branch-2.06.patch
      39 kB
      Walter Su

      Issue Links

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 6s There were no new javac warning messages.
        +1 javadoc 10m 42s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 26s The applied patch generated 44 new checkstyle issues (total was 512, now 529).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 12s Pre-build of native portion
        -1 hdfs tests 49m 59s Tests failed in hadoop-hdfs.
            96m 48s  



        Reason Tests
        FindBugs module:hadoop-hdfs
        Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.namenode.TestCommitBlockSynchronization
          hadoop.hdfs.util.TestByteArrayManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12767004/HDFS-9255.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / cf23f2c
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13026/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13026/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 42s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 26s The applied patch generated 44 new checkstyle issues (total was 512, now 529). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 49m 59s Tests failed in hadoop-hdfs.     96m 48s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestCommitBlockSynchronization   hadoop.hdfs.util.TestByteArrayManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767004/HDFS-9255.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cf23f2c Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13026/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13026/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13026/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 54s There were no new javac warning messages.
        +1 javadoc 10m 33s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 25s The applied patch generated 8 new checkstyle issues (total was 512, now 493).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        -1 findbugs 2m 34s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 13s Pre-build of native portion
        -1 hdfs tests 50m 53s Tests failed in hadoop-hdfs.
            97m 15s  



        Reason Tests
        FindBugs module:hadoop-hdfs
        Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12767097/HDFS-9255.03.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 52ac73f
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13033/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13033/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 54s There were no new javac warning messages. +1 javadoc 10m 33s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 8 new checkstyle issues (total was 512, now 493). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 34s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 13s Pre-build of native portion -1 hdfs tests 50m 53s Tests failed in hadoop-hdfs.     97m 15s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767097/HDFS-9255.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 52ac73f Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13033/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13033/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13033/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Walter Su. Nice work!

        I've few comments, please see

        1. In unit tests probably you could use GenericTestUtils.assertExceptionContains for asserting the exception messages rather than swallow it.
          +      try {
          +        spyTask.recover();
          +      } catch(IOException e){
          +        // IOException: All datanodes failed
          +      }
          
        2. class RecoveryTask does not need to be public. Also, how about renaming the class to RecoveryTaskContiguous?
        3. Do you have any plans to incorporate the comments discussed in HDFS-9173 here in this jira?
        4. Please take care the checkstyle warnings related to the patch.
        Show
        rakeshr Rakesh R added a comment - Thanks Walter Su . Nice work! I've few comments, please see In unit tests probably you could use GenericTestUtils.assertExceptionContains for asserting the exception messages rather than swallow it. + try { + spyTask.recover(); + } catch (IOException e){ + // IOException: All datanodes failed + } class RecoveryTask does not need to be public. Also, how about renaming the class to RecoveryTaskContiguous? Do you have any plans to incorporate the comments discussed in HDFS-9173 here in this jira? Please take care the checkstyle warnings related to the patch.
        Hide
        rakeshr Rakesh R added a comment -

        Adding one more comment,

        • Please add @InterfaceAudience.Private to the newly added BlockRecoveryWorker class.
        Show
        rakeshr Rakesh R added a comment - Adding one more comment, Please add @InterfaceAudience.Private to the newly added BlockRecoveryWorker class.
        Hide
        walter.k.su Walter Su added a comment -

        Thanks Rakesh R! Uploaded 04 patch address all your comments.

        Many checkstyle issues already exist before. I'll fix them when I move the code.

        Show
        walter.k.su Walter Su added a comment - Thanks Rakesh R ! Uploaded 04 patch address all your comments. Many checkstyle issues already exist before. I'll fix them when I move the code.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 19m 48s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 9m 4s There were no new javac warning messages.
        +1 javadoc 11m 58s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 36s The applied patch generated 3 new checkstyle issues (total was 513, now 489).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 38s mvn install still works.
        +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
        -1 findbugs 2m 57s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 40s Pre-build of native portion
        -1 hdfs tests 70m 56s Tests failed in hadoop-hdfs.
            122m 46s  



        Reason Tests
        FindBugs module:hadoop-hdfs
        Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
          hadoop.hdfs.TestRecoverStripedFile
          hadoop.hdfs.server.namenode.TestFSImageWithSnapshot



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12767274/HDFS-9255.04.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 476a251
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13045/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13045/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 48s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 4s There were no new javac warning messages. +1 javadoc 11m 58s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 36s The applied patch generated 3 new checkstyle issues (total was 513, now 489). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. -1 findbugs 2m 57s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 40s Pre-build of native portion -1 hdfs tests 70m 56s Tests failed in hadoop-hdfs.     122m 46s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767274/HDFS-9255.04.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 476a251 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13045/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13045/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13045/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Walter for the work. The 04 patch looks good overall. Some minors:

        1. Datanode#recoverBlocks doesn't need a return value. We can also consider replacing the method with a getter for blockRecoveryWorker and getBlockRecoveryWorker().recoverBlocks.
        2. The structure of BlockRecoveryWorker can also be simplified. Following the example of ErasureCodingWorker we can make RecoveryTaskContiguous itself a daemon. But since this is just a refactor JIRA, we can address this issue and the previous one in a separate JIRA.
        3. Since we are explicitly naming RecoveryTaskContiguous, should also throw unsupported exception for striped blocks?
          +        for(RecoveringBlock b : blocks) {
          +          try {
          +            logRecoverBlock(who, b);
          +            RecoveryTaskContiguous task = new RecoveryTaskContiguous(b);
          +            task.recover();
          
        4. I guess changes in PBHelper, DatanodeManager, and FSNamesystem are all misc optimizations not directly related to this refactor? PBHelper and DatanodeManager changes LGTM. But I haven't fully reviewed FSNamesystem. It looks a decent size optimization by itself. How about do it separately?
        Show
        zhz Zhe Zhang added a comment - Thanks Walter for the work. The 04 patch looks good overall. Some minors: Datanode#recoverBlocks doesn't need a return value. We can also consider replacing the method with a getter for blockRecoveryWorker and getBlockRecoveryWorker().recoverBlocks . The structure of BlockRecoveryWorker can also be simplified. Following the example of ErasureCodingWorker we can make RecoveryTaskContiguous itself a daemon. But since this is just a refactor JIRA, we can address this issue and the previous one in a separate JIRA. Since we are explicitly naming RecoveryTaskContiguous , should also throw unsupported exception for striped blocks? + for (RecoveringBlock b : blocks) { + try { + logRecoverBlock(who, b); + RecoveryTaskContiguous task = new RecoveryTaskContiguous(b); + task.recover(); I guess changes in PBHelper , DatanodeManager , and FSNamesystem are all misc optimizations not directly related to this refactor? PBHelper and DatanodeManager changes LGTM. But I haven't fully reviewed FSNamesystem . It looks a decent size optimization by itself. How about do it separately?
        Hide
        walter.k.su Walter Su added a comment -

        The changes in commitBlockSynchronization is to utilize DatanodeManager.getDatanodeStorageInfos(..) which knows DatanodeID.EMPTY_DATANODE_ID. But in fact EMPTY_DATANODE_ID only used for striped file.
        I meant to separte all non-related changes of HDFS-9173 to here. Sorry for the confusing, I still hope to include the changes in here.

        Show
        walter.k.su Walter Su added a comment - The changes in commitBlockSynchronization is to utilize DatanodeManager.getDatanodeStorageInfos(..) which knows DatanodeID.EMPTY_DATANODE_ID . But in fact EMPTY_DATANODE_ID only used for striped file. I meant to separte all non-related changes of HDFS-9173 to here. Sorry for the confusing, I still hope to include the changes in here.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for the clarification Walter. IIUC DatanodeManager#getDatanode(DatanodeID nodeID) returns null for EMPTY_DATANODE_ID, and that should be trimmed in trimmedTargets and trimmedStorages right? In your updated logic, nothing will be done to null entries in trimmedStorages anyway. Do we still need the change?

        Some nits if we decide to do the above change:

        1. The current logic is not trimming null entries from newtargets so trimmedStorages needs a better name
        2. Why do we deep copy storedBlock but not truncatedBlock?
                         if(copyTruncate) {
          -                storageInfo.addBlock(truncatedBlock, truncatedBlock);
          +                trimmedStorages[i].addBlock(truncatedBlock, truncatedBlock);
                         } else {
          -                storageInfo.addBlock(storedBlock, storedBlock);
          +                Block bi = new Block(storedBlock);
          +                trimmedStorages[i].addBlock(storedBlock, bi);
                         }
          
        3. We can take the chance to change newtargets to newTargets, and change the for (int i = 0; i < trimmedStorages.length loop to foreach style.
        Show
        zhz Zhe Zhang added a comment - Thanks for the clarification Walter. IIUC DatanodeManager#getDatanode(DatanodeID nodeID) returns null for EMPTY_DATANODE_ID , and that should be trimmed in trimmedTargets and trimmedStorages right? In your updated logic, nothing will be done to null entries in trimmedStorages anyway. Do we still need the change? Some nits if we decide to do the above change: The current logic is not trimming null entries from newtargets so trimmedStorages needs a better name Why do we deep copy storedBlock but not truncatedBlock ? if (copyTruncate) { - storageInfo.addBlock(truncatedBlock, truncatedBlock); + trimmedStorages[i].addBlock(truncatedBlock, truncatedBlock); } else { - storageInfo.addBlock(storedBlock, storedBlock); + Block bi = new Block(storedBlock); + trimmedStorages[i].addBlock(storedBlock, bi); } We can take the chance to change newtargets to newTargets , and change the for (int i = 0; i < trimmedStorages.length loop to foreach style.
        Hide
        walter.k.su Walter Su added a comment -

        The old way trims the invalid targets, shrinks the array ( by using a tmp ArrayList).
        In the patch, I want to replace the invalid targets, or EMPTY targets with null, but don't shrinks the array. (So trimmedStorages is not a proper name here?)

        Short story, they are for striped block.
        Long story, UC.replicas are sorted in the order of block index, so as the newTargets sent by Client/DN. So we don't have to extend updatePipeline(..) or commitBlockSynchronization(..) protocol to send indices[]. A missing block is null in the newTargets slot, in this way we know its index.
        A little awkward here...hmm...Thanks for deeply looking. Maybe it's inappropriate to keep the changes here, and I should keep the changes in HDFS-9173, how do you think?

        Show
        walter.k.su Walter Su added a comment - The old way trims the invalid targets, shrinks the array ( by using a tmp ArrayList). In the patch, I want to replace the invalid targets, or EMPTY targets with null , but don't shrinks the array. (So trimmedStorages is not a proper name here?) Short story, they are for striped block. Long story, UC.replicas are sorted in the order of block index, so as the newTargets sent by Client/DN. So we don't have to extend updatePipeline(..) or commitBlockSynchronization(..) protocol to send indices[]. A missing block is null in the newTargets slot, in this way we know its index. A little awkward here...hmm...Thanks for deeply looking. Maybe it's inappropriate to keep the changes here, and I should keep the changes in HDFS-9173 , how do you think?
        Hide
        zhz Zhe Zhang added a comment -

        Walter Su Yes I think it makes more sense to make the commitBlockSynchronization change in HDFS-9173. The change will be easier to understand in that context.

        Show
        zhz Zhe Zhang added a comment - Walter Su Yes I think it makes more sense to make the commitBlockSynchronization change in HDFS-9173 . The change will be easier to understand in that context.
        Hide
        walter.k.su Walter Su added a comment -

        1.Datanode#recoverBlocks doesn't need a return value. We can also consider replacing the method with a getter for blockRecoveryWorker and getBlockRecoveryWorker().recoverBlocks.

        1. Done. 2. Removed FSNamesystem changes. Uploaded 05 patch.

        Show
        walter.k.su Walter Su added a comment - 1.Datanode#recoverBlocks doesn't need a return value. We can also consider replacing the method with a getter for blockRecoveryWorker and getBlockRecoveryWorker().recoverBlocks. 1. Done. 2. Removed FSNamesystem changes. Uploaded 05 patch.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 17s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 51s There were no new javac warning messages.
        +1 javadoc 10m 23s There were no new javadoc warning messages.
        +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 28s The applied patch generated 1 new checkstyle issues (total was 286, now 264).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 12s Pre-build of native portion
        -1 hdfs tests 51m 35s Tests failed in hadoop-hdfs.
            97m 58s  



        Reason Tests
        FindBugs module:hadoop-hdfs
        Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768244/HDFS-9255.05.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 124a412
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13149/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13149/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 17s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 10m 23s There were no new javadoc warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 28s The applied patch generated 1 new checkstyle issues (total was 286, now 264). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 51m 35s Tests failed in hadoop-hdfs.     97m 58s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768244/HDFS-9255.05.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 124a412 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13149/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13149/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13149/console This message was automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for working on this, Walter! The patch looks pretty good to me. One question is: since DataNode#blockRecoveryWorker is not declared as final, can we make sure the BPServiceActor thread can always see its non-null value when calling getBlockRecoveryWorker?

        Show
        jingzhao Jing Zhao added a comment - Thanks for working on this, Walter! The patch looks pretty good to me. One question is: since DataNode#blockRecoveryWorker is not declared as final, can we make sure the BPServiceActor thread can always see its non-null value when calling getBlockRecoveryWorker ?
        Hide
        walter.k.su Walter Su added a comment -

        One question is: since DataNode#blockRecoveryWorker is not declared as final, can we make sure the BPServiceActor thread can always see its non-null value when calling getBlockRecoveryWorker?

        No. uploaded 06 patch to address that.

        Show
        walter.k.su Walter Su added a comment - One question is: since DataNode#blockRecoveryWorker is not declared as final, can we make sure the BPServiceActor thread can always see its non-null value when calling getBlockRecoveryWorker? No. uploaded 06 patch to address that.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 27s There were no new javac warning messages.
        +1 javadoc 11m 34s There were no new javadoc warning messages.
        +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 32s The applied patch generated 1 new checkstyle issues (total was 286, now 264).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        -1 findbugs 2m 52s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 31s Pre-build of native portion
        +1 hdfs tests 53m 26s Tests passed in hadoop-hdfs.
            102m 37s  



        Reason Tests
        FindBugs module:hadoop-hdfs



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768675/HDFS-9255.06.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b57f08c
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13189/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13189/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 27s There were no new javac warning messages. +1 javadoc 11m 34s There were no new javadoc warning messages. +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 32s The applied patch generated 1 new checkstyle issues (total was 286, now 264). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. -1 findbugs 2m 52s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 31s Pre-build of native portion +1 hdfs tests 53m 26s Tests passed in hadoop-hdfs.     102m 37s   Reason Tests FindBugs module:hadoop-hdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768675/HDFS-9255.06.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b57f08c Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13189/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13189/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13189/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Walter for the update. +1 on the latest patch (listing a few nits that can be done as follow-on). Jing Zhao Do you have further comments?

        1. BlockRecoveryWorker#recoverBlocks doesn't need a return value
        2. Even though DN now initializes the recovery worker before block pool manager, BPOfferService can still use a null check at dn.getBlockRecoveryWorker()
        Show
        zhz Zhe Zhang added a comment - Thanks Walter for the update. +1 on the latest patch (listing a few nits that can be done as follow-on). Jing Zhao Do you have further comments? BlockRecoveryWorker#recoverBlocks doesn't need a return value Even though DN now initializes the recovery worker before block pool manager, BPOfferService can still use a null check at dn.getBlockRecoveryWorker()
        Hide
        jingzhao Jing Zhao added a comment -

        +1 on the latest patch. Thanks Walter!

        Show
        jingzhao Jing Zhao added a comment - +1 on the latest patch. Thanks Walter!
        Hide
        zhz Zhe Zhang added a comment -

        Walter Su Could you set a target version? There's a conflict when backporting to branch-2

        Show
        zhz Zhe Zhang added a comment - Walter Su Could you set a target version? There's a conflict when backporting to branch-2
        Hide
        walter.k.su Walter Su added a comment -

        Thanks Zhe Zhang, Jing Zhao for review!
        And sorry for the trouble, Zhe Zhang. Uploaded HDFS-9255-branch-2.06.patch.

        Show
        walter.k.su Walter Su added a comment - Thanks Zhe Zhang , Jing Zhao for review! And sorry for the trouble, Zhe Zhang . Uploaded HDFS-9255 -branch-2.06.patch.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Walter! I just committed to both trunk and branch-2. Thanks Rakesh R and Jing Zhao for the helpful reviews as well.

        Show
        zhz Zhe Zhang added a comment - Thanks Walter! I just committed to both trunk and branch-2. Thanks Rakesh R and Jing Zhao for the helpful reviews as well.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #8719 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8719/)
        HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8719 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8719/ ) HDFS-9255 . Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #595 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/595/)
        HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #595 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/595/ ) HDFS-9255 . Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #608 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/608/)
        HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #608 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/608/ ) HDFS-9255 . Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1331 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1331/)
        HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1331 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1331/ ) HDFS-9255 . Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2484 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2484/)
        HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2484 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2484/ ) HDFS-9255 . Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #546 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/546/)
        HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #546 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/546/ ) HDFS-9255 . Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2538 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2538/)
        HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2538 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2538/ ) HDFS-9255 . Consolidate block recovery related implementation into a (zhz: rev e287e7d14b838a866ba03d895fa35819999d7c09) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        Hide
        cnauroth Chris Nauroth added a comment -

        This patch introduced a Findbugs warning, which is tracked in HDFS-9401.

        Show
        cnauroth Chris Nauroth added a comment - This patch introduced a Findbugs warning, which is tracked in HDFS-9401 .
        Hide
        zhz Zhe Zhang added a comment -

        Good catch. Thanks Chris!

        Show
        zhz Zhe Zhang added a comment - Good catch. Thanks Chris!

          People

          • Assignee:
            walter.k.su Walter Su
            Reporter:
            walter.k.su Walter Su
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development