Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. h564-24.patch
      39 kB
      Kan Zhang
    2. h564-25.patch
      40 kB
      Kan Zhang
    3. pipelineTests.patch
      31 kB
      Hairong Kuang
    4. pipelineTests1.patch
      25 kB
      Hairong Kuang
    5. pipelineTests2.patch
      37 kB
      Hairong Kuang

      Issue Links

        Activity

        Hide
        Kan Zhang added a comment -

        I've attached 2 preliminary patches.

        h564-24.patch is a patch for the pre-append-merge trunk. This patch changes the behavior of BlockReciever.java in 2 ways. 1) When downstream error happens (ends up in handleMirrorOutError()), the receiver thread no longer interrupts the responder thread, which may lead the responder to behave as if local error has occurred and give the wrong idea to upstream node. 2) The responder will try to read all downstream statuses (up to first ERROR status) before sending its own status and forwarding others to upstream node. If the responder fails to read all downstream statuses it needs, it will mark the next downstream datanode as ERROR.

        h564-24.patch implements all the tests except 26-28 and 31-33. In the case of test 26-28, I've seen intermittent failures similar to those described in HDFS-101, i.e., when the first datanode sends all statuses to DFSClient and closes the socket, DFSClient isn't able to read those statuses and instead gets a TCP reset. As a result, DFSClient will mistakenly consider the first datanode at fault. In the case of test 31-33, the DFSClient will keep receiving seqno == -1 (keep alive) and hang.

        h564-25.patch is a quick port of h564-24.patch to the current post-append-merge trunk. Unfortunately many of the tests are failing and I may not have time to investigate it. Hopefully, someone can pick it up from here.

        Show
        Kan Zhang added a comment - I've attached 2 preliminary patches. h564-24.patch is a patch for the pre-append-merge trunk. This patch changes the behavior of BlockReciever.java in 2 ways. 1) When downstream error happens (ends up in handleMirrorOutError()), the receiver thread no longer interrupts the responder thread, which may lead the responder to behave as if local error has occurred and give the wrong idea to upstream node. 2) The responder will try to read all downstream statuses (up to first ERROR status) before sending its own status and forwarding others to upstream node. If the responder fails to read all downstream statuses it needs, it will mark the next downstream datanode as ERROR. h564-24.patch implements all the tests except 26-28 and 31-33. In the case of test 26-28, I've seen intermittent failures similar to those described in HDFS-101 , i.e., when the first datanode sends all statuses to DFSClient and closes the socket, DFSClient isn't able to read those statuses and instead gets a TCP reset. As a result, DFSClient will mistakenly consider the first datanode at fault. In the case of test 31-33, the DFSClient will keep receiving seqno == -1 (keep alive) and hang. h564-25.patch is a quick port of h564-24.patch to the current post-append-merge trunk. Unfortunately many of the tests are failing and I may not have time to investigate it. Hopefully, someone can pick it up from here.
        Hide
        Hairong Kuang added a comment -

        This patch merges Kan's patch with the trunk, with some slight changes. It also contains the patch to HDFS-101. With the fix to HDFS-101, all unit tests pass. The only problem is that the new tests run for too long time. I will try to improve their speed later.

        Show
        Hairong Kuang added a comment - This patch merges Kan's patch with the trunk, with some slight changes. It also contains the patch to HDFS-101 . With the fix to HDFS-101 , all unit tests pass. The only problem is that the new tests run for too long time. I will try to improve their speed later.
        Hide
        Konstantin Boudnik added a comment -

        +1 patch looks good.

        I've ran fault injection tests and they are passing except that I see an intermittent failure in TestFiHFlush. However, it the failures are happening in the tests which do not expect to find a pipeline when hflush() is called after a write within block boundaries. I think these three tests (hFlushFi01_a, hFlushFi02_a, hFlushFi03_a) need to be revisited.

        Show
        Konstantin Boudnik added a comment - +1 patch looks good. I've ran fault injection tests and they are passing except that I see an intermittent failure in TestFiHFlush. However, it the failures are happening in the tests which do not expect to find a pipeline when hflush() is called after a write within block boundaries. I think these three tests (hFlushFi01_a, hFlushFi02_a, hFlushFi03_a) need to be revisited.
        Hide
        Hairong Kuang added a comment -

        Here is a patch that excludes fix to HDFS-101.

        Show
        Hairong Kuang added a comment - Here is a patch that excludes fix to HDFS-101 .
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Test 17-35 are not found in pipelineTests1.patch. Forgot to add them?

        Show
        Tsz Wo Nicholas Sze added a comment - Test 17-35 are not found in pipelineTests1.patch. Forgot to add them?
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428501/pipelineTests1.patch
        against trunk revision 892941.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 24 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428501/pipelineTests1.patch against trunk revision 892941. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/154/console This message is automatically generated.
        Hide
        Hairong Kuang added a comment -

        This patch includes the new tests and they all passed.

        Show
        Hairong Kuang added a comment - This patch includes the new tests and they all passed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428669/pipelineTests2.patch
        against trunk revision 893039.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 27 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428669/pipelineTests2.patch against trunk revision 893039. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/87/console This message is automatically generated.
        Hide
        Hairong Kuang added a comment -

        The failed core test is not caused by this patch because this patch adds only fault injection tests.

        Show
        Hairong Kuang added a comment - The failed core test is not caused by this patch because this patch adds only fault injection tests.
        Hide
        Hairong Kuang added a comment -

        I've committed this. Thanks, Nicholas and Kan!

        Show
        Hairong Kuang added a comment - I've committed this. Thanks, Nicholas and Kan!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #154 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/154/)
        . Adding pipeline tests 17-35. Contributed by Nicholas, Kan, and Hairong.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #154 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/154/ ) . Adding pipeline tests 17-35. Contributed by Nicholas, Kan, and Hairong.
        Hide
        Hudson added a comment -

        Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #159 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/159/)

        Show
        Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #159 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/159/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #182 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/182/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #182 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/182/ )
        Hide
        Hudson added a comment -

        Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #94 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/94/)

        Show
        Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #94 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/94/ )

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Kan Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development