Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Added a new stage TRANSFER_RBW to DataTransferProtocol

      Description

      This is the step (*) described here.

      1. h1675_20110228.patch
        16 kB
        Tsz Wo Nicholas Sze
      2. h1675_20110228b.patch
        15 kB
        Tsz Wo Nicholas Sze
      3. h1675_20110308.patch
        20 kB
        Tsz Wo Nicholas Sze
      4. h1675_20110310.patch
        20 kB
        Tsz Wo Nicholas Sze
      5. h1675_20110310b.patch
        22 kB
        Tsz Wo Nicholas Sze
      6. h1675_20110310c.patch
        22 kB
        Tsz Wo Nicholas Sze
      7. h1675_20110311.patch
        23 kB
        Tsz Wo Nicholas Sze
      8. h1675_20110313.patch
        23 kB
        Tsz Wo Nicholas Sze

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #559 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/559/)
        HDFS-1675. Support transferring RBW between datanodes.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #559 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/559/ ) HDFS-1675 . Support transferring RBW between datanodes.
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Incompatible change, Reviewed]
        Release Note Added a new stage TRANSFER_RBW to DataTransferProtocol
        Fix Version/s 0.23.0 [ 12315571 ]
        Resolution Fixed [ 1 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Thanks Jitendra for the careful reviews!

        I have committed this.

        Show
        Tsz Wo Nicholas Sze added a comment - Thanks Jitendra for the careful reviews! I have committed this.
        Hide
        Jitendra Nath Pandey added a comment -

        +1 for the patch.

        Show
        Jitendra Nath Pandey added a comment - +1 for the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473530/h1675_20110313.patch
        against trunk revision 1080836.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestFileConcurrentReader

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/253//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/253//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/253//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473530/h1675_20110313.patch against trunk revision 1080836. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/253//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/253//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/253//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Attachment h1675_20110313.patch [ 12473530 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        h1675_20110313.patch: made the changes according to Jitendra's comments.

        Show
        Tsz Wo Nicholas Sze added a comment - h1675_20110313.patch: made the changes according to Jitendra's comments.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Thanks again Jitendra.

        > In method transferBlockForPipelineRecovery, shouldn't the DataTransfer object be created with stage = BlockConstructionStage.TRANSFER_RBW ?

        The test indeed catches it but why hudson does not?

        Testcase: testTransferRbw took 5.694 sec
        	FAILED
        expected:<TEMPORARY> but was:<FINALIZED>
        junit.framework.AssertionFailedError: expected:<TEMPORARY> but was:<FINALIZED>
        	at org.apache.hadoop.hdfs.server.datanode.TestTransferRbw.getReplica(TestTransferRbw.java:58)
        	at org.apache.hadoop.hdfs.server.datanode.TestTransferRbw.testTransferRbw(TestTransferRbw.java:115)
        
        Show
        Tsz Wo Nicholas Sze added a comment - Thanks again Jitendra. > In method transferBlockForPipelineRecovery, shouldn't the DataTransfer object be created with stage = BlockConstructionStage.TRANSFER_RBW ? The test indeed catches it but why hudson does not? Testcase: testTransferRbw took 5.694 sec FAILED expected:<TEMPORARY> but was:<FINALIZED> junit.framework.AssertionFailedError: expected:<TEMPORARY> but was:<FINALIZED> at org.apache.hadoop.hdfs.server.datanode.TestTransferRbw.getReplica(TestTransferRbw.java:58) at org.apache.hadoop.hdfs.server.datanode.TestTransferRbw.testTransferRbw(TestTransferRbw.java:115)
        Hide
        Jitendra Nath Pandey added a comment -

        In method transferBlockForPipelineRecovery, shouldn't the DataTransfer object be created with stage = BlockConstructionStage.TRANSFER_RBW ?

        In BlockReceiver.java#receiveBlock
        + if (clientName.length() == 0
        + && initialStage != BlockConstructionStage.TRANSFER_RBW) {
        No need to check for clientName.length() == 0 , it is already checked before.

        Show
        Jitendra Nath Pandey added a comment - In method transferBlockForPipelineRecovery, shouldn't the DataTransfer object be created with stage = BlockConstructionStage.TRANSFER_RBW ? In BlockReceiver.java#receiveBlock + if (clientName.length() == 0 + && initialStage != BlockConstructionStage.TRANSFER_RBW) { No need to check for clientName.length() == 0 , it is already checked before.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473452/h1675_20110311.patch
        against trunk revision 1080380.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.server.datanode.TestBlockReport
        org.apache.hadoop.hdfs.TestFileConcurrentReader

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/251//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/251//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/251//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473452/h1675_20110311.patch against trunk revision 1080380. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/251//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/251//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/251//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Attachment h1675_20110311.patch [ 12473452 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The failed tests seem related although I cannot get them fail in my machine.

        h1675_20110311.patch: Added BlockConstructionStage.TRANSFER_RBW

        Show
        Tsz Wo Nicholas Sze added a comment - The failed tests seem related although I cannot get them fail in my machine. h1675_20110311.patch: Added BlockConstructionStage.TRANSFER_RBW
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473360/h1675_20110310c.patch
        against trunk revision 1080380.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.server.balancer.TestBalancer
        org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
        org.apache.hadoop.hdfs.TestFileAppend4
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.hdfs.TestLargeBlock
        org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/249//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/249//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/249//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473360/h1675_20110310c.patch against trunk revision 1080380. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestWriteConfigurationToDFS -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/249//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/249//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/249//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Attachment h1675_20110310c.patch [ 12473360 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        h1675_20110310c.patch: changed transferRbw(..) to transferBlockForPipelineRecovery(..) for supporting the cases in the previous comment.

        Show
        Tsz Wo Nicholas Sze added a comment - h1675_20110310c.patch: changed transferRbw(..) to transferBlockForPipelineRecovery(..) for supporting the cases in the previous comment.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        In order to make sure the patch is useful for HDFS-1606, Jitendra and I came up with the following cases.

        • Case 1: Failure in Pipeline Setup
          • Append
            • Transfer the stored replica, which may be a RBW or a finalized replica.
          • Create
            • If no data, then no transfer is required.
            • If there are data written, transfer RBW. This case may happens when there are streaming failure earlier in this pipeline.
        • Case 2: Failure in Streaming or Close
          • Append/Create:
            • transfer RBW

        Therefore, a method is required to supporting transferring the stored replica, RBW or finalized, of the source datanode. It should return visible length, stored GS and whether it is a RBW.

        Show
        Tsz Wo Nicholas Sze added a comment - In order to make sure the patch is useful for HDFS-1606 , Jitendra and I came up with the following cases. Case 1: Failure in Pipeline Setup Append Transfer the stored replica, which may be a RBW or a finalized replica. Create If no data, then no transfer is required. If there are data written, transfer RBW. This case may happens when there are streaming failure earlier in this pipeline. Case 2: Failure in Streaming or Close Append/Create : transfer RBW Therefore, a method is required to supporting transferring the stored replica, RBW or finalized, of the source datanode. It should return visible length, stored GS and whether it is a RBW.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473343/h1675_20110310b.patch
        against trunk revision 1080380.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.server.balancer.TestBalancer
        org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
        org.apache.hadoop.hdfs.server.datanode.TestBlockReport
        org.apache.hadoop.hdfs.server.namenode.TestNodeCount
        org.apache.hadoop.hdfs.TestFileConcurrentReader

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/247//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/247//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/247//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473343/h1675_20110310b.patch against trunk revision 1080380. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.namenode.TestNodeCount org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/247//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/247//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/247//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1675_20110310b.patch [ 12473343 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        h1675_20110310b.patch:

        • uses numBytes instead of bytesOnDisk
        • should not send block receive to NN
        • get newrbw from newnode in the test.
        Show
        Tsz Wo Nicholas Sze added a comment - h1675_20110310b.patch: uses numBytes instead of bytesOnDisk should not send block receive to NN get newrbw from newnode in the test.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473325/h1675_20110310.patch
        against trunk revision 1080095.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
        org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
        org.apache.hadoop.hdfs.TestFileConcurrentReader

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/246//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/246//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/246//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473325/h1675_20110310.patch against trunk revision 1080095. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/246//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/246//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/246//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Attachment h1675_20110310.patch [ 12473325 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        h1675_20110310.patch:

        • first temporary and then RBW
        • reverted the protocol changes (will do it later)
        Show
        Tsz Wo Nicholas Sze added a comment - h1675_20110310.patch: first temporary and then RBW reverted the protocol changes (will do it later)
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > I feel that it is better to put RBW in tmp first and then move to rbw when copy is done. ...

        You are right. It would align with replications (i.e. first temporary and then finalize).

        Show
        Tsz Wo Nicholas Sze added a comment - > I feel that it is better to put RBW in tmp first and then move to rbw when copy is done. ... You are right. It would align with replications (i.e. first temporary and then finalize).
        Hide
        Hairong Kuang added a comment -

        I feel that it is better to put RBW in tmp first and then move to rbw when copy is done. Otherwise, if there is an error during copy then copy aborts, lease recovery will make a wrong decision deciding replica length and then truncate visible bytes.

        Show
        Hairong Kuang added a comment - I feel that it is better to put RBW in tmp first and then move to rbw when copy is done. Otherwise, if there is an error during copy then copy aborts, lease recovery will make a wrong decision deciding replica length and then truncate visible bytes.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473086/h1675_20110308.patch
        against trunk revision 1079069.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:

        -1 contrib tests. The patch failed contrib unit tests.

        -1 system test framework. The patch failed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/243//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/243//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/243//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473086/h1675_20110308.patch against trunk revision 1079069. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/243//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/243//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/243//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Attachment h1675_20110308.patch [ 12473086 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > ... Need sometime to refresh my memory of what we did in the append project.

        Hairong, the append code is easy to understand since you have added a lot of comments. Great works!

        h1675_20110308.patch

        • creating RBW, instead of temporary, in destination
        • updated DataTransferProtocol and ClientDataNodeProtocol
        • fixed the unit test
        Show
        Tsz Wo Nicholas Sze added a comment - > ... Need sometime to refresh my memory of what we did in the append project. Hairong, the append code is easy to understand since you have added a lot of comments. Great works! h1675_20110308.patch creating RBW, instead of temporary, in destination updated DataTransferProtocol and ClientDataNodeProtocol fixed the unit test
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > In the test, after the call to transferRbw, we should also compare the rbw on the new node.

        Jitendra, this is a good catch! Thanks. It turns out that the replica created in destination is a temporary but not a RBW. Need some works.

        > Overall, it looks good. So RBW will be copied to the tmp directory at the receiver side as what finalized replicas are replicated, right? I think it should get moved to the rbw directory and rbw map after copy is done.

        Hairong, thanks for taking a look. The idea is to create a RBW in destination but not finalize it. So no further move is required.

        Show
        Tsz Wo Nicholas Sze added a comment - > In the test, after the call to transferRbw, we should also compare the rbw on the new node. Jitendra, this is a good catch! Thanks. It turns out that the replica created in destination is a temporary but not a RBW. Need some works. > Overall, it looks good. So RBW will be copied to the tmp directory at the receiver side as what finalized replicas are replicated, right? I think it should get moved to the rbw directory and rbw map after copy is done. Hairong, thanks for taking a look. The idea is to create a RBW in destination but not finalize it. So no further move is required.
        Hide
        Hairong Kuang added a comment -

        Hi Nicolas, sorry for my delay at reviewing this patch. Need sometime to refresh my memory of what we did in the append project.

        Overall, it looks good. So RBW will be copied to the tmp directory at the receiver side as what finalized replicas are replicated, right? I think it should get moved to the rbw directory and rbw map after copy is done.

        Show
        Hairong Kuang added a comment - Hi Nicolas, sorry for my delay at reviewing this patch. Need sometime to refresh my memory of what we did in the append project. Overall, it looks good. So RBW will be copied to the tmp directory at the receiver side as what finalized replicas are replicated, right? I think it should get moved to the rbw directory and rbw map after copy is done.
        Hide
        Jitendra Nath Pandey added a comment -

        In the test, after the call to transferRbw, we should also compare the rbw on the new node.

        Show
        Jitendra Nath Pandey added a comment - In the test, after the call to transferRbw, we should also compare the rbw on the new node.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472274/h1675_20110228b.patch
        against trunk revision 1075571.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/220//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/220//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/220//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472274/h1675_20110228b.patch against trunk revision 1075571. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/220//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/220//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/220//console This message is automatically generated.
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1675_20110228b.patch [ 12472274 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        h1675_20110228b.patch: I should not removed SimulatedFSDataset() ... Uggg

        This passed all unit tests except TestDataNodeMXBean (HDFS-1625).

        Show
        Tsz Wo Nicholas Sze added a comment - h1675_20110228b.patch: I should not removed SimulatedFSDataset() ... Uggg This passed all unit tests except TestDataNodeMXBean ( HDFS-1625 ).
        Tsz Wo Nicholas Sze made changes -
        Field Original Value New Value
        Attachment h1675_20110228.patch [ 12472268 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        h1675_20110228.patch:

        • changed DataNode.DataTransfer and BlockSender to support transferring RBW.
        • added FSDatasetInterface.isValidRbw(..).
        • added a new test, TestTransferRbw.
        Show
        Tsz Wo Nicholas Sze added a comment - h1675_20110228.patch: changed DataNode.DataTransfer and BlockSender to support transferring RBW. added FSDatasetInterface.isValidRbw(..) . added a new test, TestTransferRbw .
        Tsz Wo Nicholas Sze created issue -

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development