Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This JIRA is created to be a place holder for the test implemented as a part of HDFS-483 fix

      1. HDFS-521.patch
        30 kB
        Konstantin Boudnik
      2. HDFS-521.patch
        30 kB
        Konstantin Boudnik
      3. HDFS-521.patch
        30 kB
        Konstantin Boudnik
      4. HDFS-521.patch
        36 kB
        Konstantin Boudnik
      5. HDFS-521.patch
        36 kB
        Konstantin Boudnik
      6. HDFS-521.patch
        36 kB
        Konstantin Boudnik
      7. HDFS-521.patch
        38 kB
        Konstantin Boudnik
      8. HDFS-521.patch
        29 kB
        Konstantin Boudnik
      9. HDFS-521.patch
        27 kB
        Konstantin Boudnik
      10. HDFS-521.patch
        27 kB
        Konstantin Boudnik
      11. HDFS-521.patch
        22 kB
        Konstantin Boudnik
      12. hdfs-521.patch
        21 kB
        Konstantin Boudnik
      13. hdfs-521.patch
        21 kB
        Konstantin Boudnik
      14. hdfs-521.patch
        16 kB
        Konstantin Boudnik
      15. HDFS-521.patch
        5 kB
        Konstantin Boudnik
      16. HDFS-521.patch
        13 kB
        Konstantin Boudnik
      17. HDFS-521.patch
        3 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Normal pipeline test cases

          Show
          Konstantin Boudnik added a comment - Normal pipeline test cases
          Hide
          Konstantin Boudnik added a comment -

          Test case placeholders

          Show
          Konstantin Boudnik added a comment - Test case placeholders
          Hide
          Konstantin Boudnik added a comment -

          New aspects are developed, some test cases are doing what they suppose to do.
          I need to find out what datanode 'successful' state means and how to correctly verify if a packet has been placed into a disk file or an internal buffer.

          Show
          Konstantin Boudnik added a comment - New aspects are developed, some test cases are doing what they suppose to do. I need to find out what datanode 'successful' state means and how to correctly verify if a packet has been placed into a disk file or an internal buffer.
          Hide
          Konstantin Boudnik added a comment -

          Without implementation for HDFS-608 it seems to be wasteful to implement these tests - too much effort is needed to access the needed packet's information. Besides, all that effort will be trashed as soon as the blocker is resolved.

          Thus, the implementation of the JIRA is postponed until after 0.21 time frame

          Show
          Konstantin Boudnik added a comment - Without implementation for HDFS-608 it seems to be wasteful to implement these tests - too much effort is needed to access the needed packet's information. Besides, all that effort will be trashed as soon as the blocker is resolved. Thus, the implementation of the JIRA is postponed until after 0.21 time frame
          Hide
          Konstantin Boudnik added a comment -

          After some reconsideration and Hairong's comments the original tests scenarios look too complicated. This is the first version of new tests implementation. Test plan description of the test cases will be updated accordingly.

          Show
          Konstantin Boudnik added a comment - After some reconsideration and Hairong's comments the original tests scenarios look too complicated. This is the first version of new tests implementation. Test plan description of the test cases will be updated accordingly.
          Hide
          Konstantin Boudnik added a comment -

          Initial version of two more test cases are added. These are using injection to verify number of bytes received and acknowledged by datanodes. Because it uses injection framework the class has been moved to src/test/aop hierarchy.

          One of the problems with the new test cases is that the acknowledgment verification happens only for two nodes - it doesn't happen for the last DN in a pipeline. Perhaps the binding of the aspect needs a fine tuning.

          Show
          Konstantin Boudnik added a comment - Initial version of two more test cases are added. These are using injection to verify number of bytes received and acknowledged by datanodes. Because it uses injection framework the class has been moved to src/test/aop hierarchy. One of the problems with the new test cases is that the acknowledgment verification happens only for two nodes - it doesn't happen for the last DN in a pipeline. Perhaps the binding of the aspect needs a fine tuning.
          Hide
          Konstantin Boudnik added a comment -

          All test cases are pretty much done, except that I'm not sure how to automate the verification of the very last test case. Namely, it seems to be tricky to findout when DFS client gives up on waiting. Need dig a bit more into this

          Test method in TestReadWhileWriting has been renamed to Pipeline_02_03 to illustrate that this is a test from pipeline test plan.

          Show
          Konstantin Boudnik added a comment - All test cases are pretty much done, except that I'm not sure how to automate the verification of the very last test case. Namely, it seems to be tricky to findout when DFS client gives up on waiting. Need dig a bit more into this Test method in TestReadWhileWriting has been renamed to Pipeline_02_03 to illustrate that this is a test from pipeline test plan.
          Hide
          Konstantin Boudnik added a comment -

          Sync'd with trunk after HDFS-716 has been committed.
          The first test case is failing intermittently because 'all datanodes are bad' diagnostics

          Show
          Konstantin Boudnik added a comment - Sync'd with trunk after HDFS-716 has been committed. The first test case is failing intermittently because 'all datanodes are bad' diagnostics
          Hide
          Konstantin Boudnik added a comment -

          The last test case verifies if more than 81 packets were sent, which seems to be non-expected behavior

          Show
          Konstantin Boudnik added a comment - The last test case verifies if more than 81 packets were sent, which seems to be non-expected behavior
          Hide
          Konstantin Boudnik added a comment -

          Merged with trunk.
          Write blocking is 100% reproducible, however even passing acks won't unblock it.

          Show
          Konstantin Boudnik added a comment - Merged with trunk. Write blocking is 100% reproducible, however even passing acks won't unblock it.
          Hide
          Konstantin Boudnik added a comment -

          test case method names are change to comply with Java style.

          Show
          Konstantin Boudnik added a comment - test case method names are change to comply with Java style.
          Hide
          Konstantin Boudnik added a comment -

          It turns out that shutting down the mini cluster solve the test's automation problem.
          Everything seems be working now.

          Show
          Konstantin Boudnik added a comment - It turns out that shutting down the mini cluster solve the test's automation problem. Everything seems be working now.
          Hide
          Konstantin Boudnik added a comment -

          It turned out that TestReadWhileWriting had windows type of line ends. It breaks the patch process big time. However tools like IntelliJ IDEA and Eclipse are able to take care about this

          Otherwise, patch is exactly as the previous one.

          Show
          Konstantin Boudnik added a comment - It turned out that TestReadWhileWriting had windows type of line ends. It breaks the patch process big time. However tools like IntelliJ IDEA and Eclipse are able to take care about this Otherwise, patch is exactly as the previous one.
          Hide
          Konstantin Boudnik added a comment -

          Merged with the latest trunk: GenericTestUtils is no longer needed for HDFS-616 is committed.

          Show
          Konstantin Boudnik added a comment - Merged with the latest trunk: GenericTestUtils is no longer needed for HDFS-616 is committed.
          Hide
          Konstantin Boudnik added a comment -

          I have ran fault injection tests and local test-patch. All's well.

          There appear to be 105 release audit warnings before the patch and 105 release audit warnings after applying the patch.
          
          +1 overall.
              +1 @author.  The patch does not contain any @author tags.
              +1 tests included.  The patch appears to include 15 new or modified tests.
              +1 javadoc.  The javadoc tool did not generate any warning messages.
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
              +1 findbugs.  The patch does not introduce any new Findbugs warnings.
              +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Konstantin Boudnik added a comment - I have ran fault injection tests and local test-patch. All's well. There appear to be 105 release audit warnings before the patch and 105 release audit warnings after applying the patch. +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Hairong Kuang added a comment -

          I checked all non-fi related changes. All look good.

          Show
          Hairong Kuang added a comment - I checked all non-fi related changes. All look good.
          Hide
          Konstantin Boudnik added a comment -

          Seems like the patch is ready for prime time.

          Show
          Konstantin Boudnik added a comment - Seems like the patch is ready for prime time.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423081/HDFS-521.patch
          against trunk revision 829258.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423081/HDFS-521.patch against trunk revision 829258. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/60/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Fixes cast issue in some of the tests.

          Show
          Konstantin Boudnik added a comment - Fixes cast issue in some of the tests.
          Hide
          Konstantin Boudnik added a comment -

          There was an apparent problem with a casting in some of the test.

          Show
          Konstantin Boudnik added a comment - There was an apparent problem with a casting in some of the test.
          Hide
          Konstantin Boudnik added a comment -

          All tests are ran locally and have finished successfully.

          Show
          Konstantin Boudnik added a comment - All tests are ran locally and have finished successfully.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423219/HDFS-521.patch
          against trunk revision 829258.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423219/HDFS-521.patch against trunk revision 829258. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/76/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Looks like that proposed multiple write doesn't work as expected. Returning back to the original single write scenario.

          Show
          Konstantin Boudnik added a comment - Looks like that proposed multiple write doesn't work as expected. Returning back to the original single write scenario.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Cos, are you changing anything in TestReadWhileWriting? The entire file TestReadWhileWriting.java was replaced. I cannot tell whether there is anything changed.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Cos, are you changing anything in TestReadWhileWriting? The entire file TestReadWhileWriting.java was replaced. I cannot tell whether there is anything changed.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Look like the patch is using the fault-injection framework to implement non-fault-injection tests. I am not sure whether it is a good idea because it is at least confusing.

          Show
          Tsz Wo Nicholas Sze added a comment - Look like the patch is using the fault-injection framework to implement non-fault-injection tests. I am not sure whether it is a good idea because it is at least confusing.
          Hide
          Konstantin Boudnik added a comment -

          All lines of the file were having windows-like line termination ^M. I've fixed that. Other than that there's a test method name change to be in synch with append test plan.

          Show
          Konstantin Boudnik added a comment - All lines of the file were having windows-like line termination ^M . I've fixed that. Other than that there's a test method name change to be in synch with append test plan.
          Hide
          Konstantin Boudnik added a comment -

          Actually it is using 'injection' framework in a broader sense. Also, it creates faults in the system to prevent acks from being delivered and such. So, I don't see any confusion there.

          Show
          Konstantin Boudnik added a comment - Actually it is using 'injection' framework in a broader sense. Also, it creates faults in the system to prevent acks from being delivered and such. So, I don't see any confusion there.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423229/HDFS-521.patch
          against trunk revision 829894.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423229/HDFS-521.patch against trunk revision 829894. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/77/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Test failures are unrelated and caused by HDFS-733

          Show
          Konstantin Boudnik added a comment - Test failures are unrelated and caused by HDFS-733
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I am neutral on using AspectJ to implement tests.

          The confusion is that we are using "fault-injection", "fi" or similar terms everywhere. Even the class of the new non-fault-injection tests in the patch is called TestFiPipelines. It seems breaking the naming convention.

          Show
          Tsz Wo Nicholas Sze added a comment - I am neutral on using AspectJ to implement tests. The confusion is that we are using "fault-injection", "fi" or similar terms everywhere. Even the class of the new non-fault-injection tests in the patch is called TestFiPipelines. It seems breaking the naming convention.
          Hide
          Konstantin Boudnik added a comment -

          Oh, I see your point. I'll rename this particular class and will move it out of o.a.h.fi package.
          Also, the test class mixes fault-injection and non-fault-injection tests together. Do you think it'd be more appropriate to split them into two different test classes?

          Show
          Konstantin Boudnik added a comment - Oh, I see your point. I'll rename this particular class and will move it out of o.a.h.fi package. Also, the test class mixes fault-injection and non-fault-injection tests together. Do you think it'd be more appropriate to split them into two different test classes?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > All lines of the file were having windows-like line termination ^M. I've fixed that. Other than that there's a test method name change to be in synch with append test plan.

          Could you fix the ^M problem in a separated issue? This problem was introduced by the patch committed in HDFS-570. There are also a few other files having the same problem.

          Show
          Tsz Wo Nicholas Sze added a comment - > All lines of the file were having windows-like line termination ^M. I've fixed that. Other than that there's a test method name change to be in synch with append test plan. Could you fix the ^M problem in a separated issue? This problem was introduced by the patch committed in HDFS-570 . There are also a few other files having the same problem.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Also, the test class mixes fault-injection and non-fault-injection tests together. Do you think it'd be more appropriate to split them into two different test classes?

          I think we need change the naming convention including the ant targets (e.g. compile-fault-inject, run-test-hdfs-fault-inject, etc), test class name (TestFiXxx), build-fi directory, etc. We may change them in a separated issue.

          Show
          Tsz Wo Nicholas Sze added a comment - > Also, the test class mixes fault-injection and non-fault-injection tests together. Do you think it'd be more appropriate to split them into two different test classes? I think we need change the naming convention including the ant targets (e.g. compile-fault-inject, run-test-hdfs-fault-inject, etc), test class name (TestFiXxx), build-fi directory, etc. We may change them in a separated issue.
          Hide
          Konstantin Boudnik added a comment -

          I do respectfully disagree that we need to change all of it for a simple reason of using AspectJ both for fault- and injection purposes. If I've understood your proposal correctly, you want to have a separate set of everything for fault-injection and a 'normal injection' (for a lack of better word). right? I think it's too excessive to have 2 different sets of instrumented code.

          I agree that 'fault-injection' isn't a best name for what the framework is clearly involving into... So we might consider this change later on. I think it's simpler and more appropriate to split non-fault-injecting and fault-injecting tests into different classes and name test methods according to the test plan.

          Show
          Konstantin Boudnik added a comment - I do respectfully disagree that we need to change all of it for a simple reason of using AspectJ both for fault- and injection purposes. If I've understood your proposal correctly, you want to have a separate set of everything for fault-injection and a 'normal injection' (for a lack of better word). right? I think it's too excessive to have 2 different sets of instrumented code. I agree that 'fault-injection' isn't a best name for what the framework is clearly involving into... So we might consider this change later on. I think it's simpler and more appropriate to split non-fault-injecting and fault-injecting tests into different classes and name test methods according to the test plan.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... you want to have a separate set of everything for fault-injection and a 'normal injection' (for a lack of better word). right? ...

          No, I mean renaming the term "fault-injection" to something else, say "aop", but not adding another framework.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... you want to have a separate set of everything for fault-injection and a 'normal injection' (for a lack of better word). right? ... No, I mean renaming the term "fault-injection" to something else, say "aop", but not adding another framework.
          Hide
          Konstantin Boudnik added a comment -

          After the discussion with Nicholas I'm going to split the tests cases into two different classes and rename some of the utility ones to make it more clearer how injection framework is used for non-fault-injected tests.

          Show
          Konstantin Boudnik added a comment - After the discussion with Nicholas I'm going to split the tests cases into two different classes and rename some of the utility ones to make it more clearer how injection framework is used for non-fault-injected tests.
          Hide
          Konstantin Boudnik added a comment -

          I've ran test-patch locally:

          There appear to be 105 release audit warnings before the patch and 105 release audit warnings after applying the patch.
          
          +1 overall.
              +1 @author.  The patch does not contain any @author tags.
              +1 tests included.  The patch appears to include 18 new or modified tests.
              +1 javadoc.  The javadoc tool did not generate any warning messages.
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
              +1 findbugs.  The patch does not introduce any new Findbugs warnings.
              +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Also, test were executed successfully:

          % ant run-test-hdfs-fault-inject
          run-test-hdfs:
          ...
              [junit] Running org.apache.hadoop.hdfs.TestFiHFlush
              [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 35.187 sec
              [junit] Running org.apache.hadoop.hdfs.TestFiPipelines
              [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 13.595 sec
              [junit] Running org.apache.hadoop.hdfs.server.datanode.TestFiDataTransferProtocol
              [junit] Tests run: 29, Failures: 0, Errors: 0, Time elapsed: 355.143 sec
          
          
          % ant run-test-hdfs -Dtestcase=TestPipelines
          run-test-hdfs:
          ...
              [junit] Running org.apache.hadoop.hdfs.TestPipelines
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.662 sec
          
          
          % ant run-test-hdfs -Dtestcase=TestReadWhileWriting
          run-test-hdfs:
          ...
              [junit] Running org.apache.hadoop.hdfs.TestReadWhileWriting
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.735 sec
          
          

          }

          Show
          Konstantin Boudnik added a comment - I've ran test-patch locally: There appear to be 105 release audit warnings before the patch and 105 release audit warnings after applying the patch. +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. Also, test were executed successfully: % ant run-test-hdfs-fault-inject run-test-hdfs: ... [junit] Running org.apache.hadoop.hdfs.TestFiHFlush [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 35.187 sec [junit] Running org.apache.hadoop.hdfs.TestFiPipelines [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 13.595 sec [junit] Running org.apache.hadoop.hdfs.server.datanode.TestFiDataTransferProtocol [junit] Tests run: 29, Failures: 0, Errors: 0, Time elapsed: 355.143 sec % ant run-test-hdfs -Dtestcase=TestPipelines run-test-hdfs: ... [junit] Running org.apache.hadoop.hdfs.TestPipelines [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.662 sec % ant run-test-hdfs -Dtestcase=TestReadWhileWriting run-test-hdfs: ... [junit] Running org.apache.hadoop.hdfs.TestReadWhileWriting [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.735 sec }
          Hide
          Konstantin Boudnik added a comment -

          Here's new version of the patch for the last run of tests and test-patch

          Show
          Konstantin Boudnik added a comment - Here's new version of the patch for the last run of tests and test-patch
          Hide
          Konstantin Boudnik added a comment -

          Seems to be good to go. Reviewers' comments are addressed.

          Show
          Konstantin Boudnik added a comment - Seems to be good to go. Reviewers' comments are addressed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423263/HDFS-521.patch
          against trunk revision 830003.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423263/HDFS-521.patch against trunk revision 830003. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/79/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Race condition between main and external checker thread.

          Show
          Konstantin Boudnik added a comment - Race condition between main and external checker thread.
          Hide
          Konstantin Boudnik added a comment -

          Fixing the race condition between two thread:

          • external thread tries to shutdown the cluster when a condition is met
          • main thread invokes @After method which checks cluster reference for null and then shuts down datanodes and then the cluster itself. The cluster references might become null in between of these two calls. Thus a separate check for null is in order.
          Show
          Konstantin Boudnik added a comment - Fixing the race condition between two thread: external thread tries to shutdown the cluster when a condition is met main thread invokes @After method which checks cluster reference for null and then shuts down datanodes and then the cluster itself. The cluster references might become null in between of these two calls. Thus a separate check for null is in order.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423328/HDFS-521.patch
          against trunk revision 830003.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423328/HDFS-521.patch against trunk revision 830003. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/80/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Ok, the failing test is because of HDFS-733. The rest looks good.

          Show
          Konstantin Boudnik added a comment - Ok, the failing test is because of HDFS-733 . The rest looks good.
          Hide
          Hairong Kuang added a comment -

          1. miss testcase pipeline-03.
          2. for pipeline-04 and 05, should check br and ba on all datanodes in the pipeline
          3. for pipeline-06, no need of wait in QueueChecker after the queue is filled up.

          Show
          Hairong Kuang added a comment - 1. miss testcase pipeline-03. 2. for pipeline-04 and 05, should check br and ba on all datanodes in the pipeline 3. for pipeline-06, no need of wait in QueueChecker after the queue is filled up.
          Hide
          Konstantin Boudnik added a comment -

          Addressing Hairong's comments:

          • datanodes order is now preserved by using a List instead of HashMap
          • the invariant is verified across all datanodes
          • redundant wait is removed
          • testcase _03 is suppose to be covered by TestReadWhileWriting however a multiple packet writes creates some strange problems in the test, which might be taken care off separately
          Show
          Konstantin Boudnik added a comment - Addressing Hairong's comments: datanodes order is now preserved by using a List instead of HashMap the invariant is verified across all datanodes redundant wait is removed testcase _03 is suppose to be covered by TestReadWhileWriting however a multiple packet writes creates some strange problems in the test, which might be taken care off separately
          Hide
          Konstantin Boudnik added a comment -

          Resubmitting the patch for another round of verification after modifying it.

          Show
          Konstantin Boudnik added a comment - Resubmitting the patch for another round of verification after modifying it.
          Hide
          Hairong Kuang added a comment -

          +1

          > testcase _03 is suppose to be covered by TestReadWhileWriting however a multiple packet writes creates some strange problems in the test, which might be taken care off separately
          Could you please open a jira on this? Is it a problem with the test or the pipeline implementation?

          Show
          Hairong Kuang added a comment - +1 > testcase _03 is suppose to be covered by TestReadWhileWriting however a multiple packet writes creates some strange problems in the test, which might be taken care off separately Could you please open a jira on this? Is it a problem with the test or the pipeline implementation?
          Hide
          Konstantin Boudnik added a comment -

          Thanks for the review, Hairong. This new JIRA is about this particular issue.

          Show
          Konstantin Boudnik added a comment - Thanks for the review, Hairong. This new JIRA is about this particular issue.
          Hide
          Konstantin Boudnik added a comment -

          I've just committed this.

          Show
          Konstantin Boudnik added a comment - I've just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #96 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/96/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #96 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/96/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/136/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #136 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/136/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/ )

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development