Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-451

Test DataTransferProtocol with fault injection

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DataTransferProtocol is a complicated protocol. It deserves to have some fault injection tests.

      1. h451_20090806.patch
        25 kB
        Tsz Wo Nicholas Sze
      2. h451_20090807.patch
        24 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          20h 34m 1 Tsz Wo Nicholas Sze 07/Aug/09 19:42
          Open Open Patch Available Patch Available
          38d 47m 2 Tsz Wo Nicholas Sze 07/Aug/09 19:44
          Patch Available Patch Available Resolved Resolved
          4h 35m 1 Tsz Wo Nicholas Sze 08/Aug/09 00:19
          Resolved Resolved Closed Closed
          381d 21h 29m 1 Tom White 24/Aug/10 21:48
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #47 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/47/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #47 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/47/ )
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this.
          Hide
          Konstantin Boudnik added a comment -

          > This might be caused by the fact that all fault injection tests are executed (as well as their results are collected) under build-fi/, which differs from the standard location.

          Then, how could we make Hudson working with build-fi?

          I believe it suppose to be done on Hudson side of which I know very little. Lemme find out and then make it happen.

          Show
          Konstantin Boudnik added a comment - > This might be caused by the fact that all fault injection tests are executed (as well as their results are collected) under build-fi/, which differs from the standard location. Then, how could we make Hudson working with build-fi? I believe it suppose to be done on Hudson side of which I know very little. Lemme find out and then make it happen.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > This might be caused by the fact that all fault injection tests are executed (as well as their results are collected) under build-fi/, which differs from the standard location.
          Then, how could we make Hudson working with build-fi?

          > Also, I'd suggest this (writing new tests for JUnit4 instead of JUnit3)
          Since the patch just have passed Hudson and I am still working on more tests, let do the junit4 changes in a later issue.

          Show
          Tsz Wo Nicholas Sze added a comment - > This might be caused by the fact that all fault injection tests are executed (as well as their results are collected) under build-fi/, which differs from the standard location. Then, how could we make Hudson working with build-fi? > Also, I'd suggest this (writing new tests for JUnit4 instead of JUnit3) Since the patch just have passed Hudson and I am still working on more tests, let do the junit4 changes in a later issue.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12415871/h451_20090807.patch
          against trunk revision 801736.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415871/h451_20090807.patch against trunk revision 801736. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/48/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Also, I'd suggest this (writing new tests for JUnit4 instead of JUnit3)

          Show
          Konstantin Boudnik added a comment - Also, I'd suggest this (writing new tests for JUnit4 instead of JUnit3)
          Hide
          Konstantin Boudnik added a comment -

          Note that the fault inject tests do not show up in the test result page although they have been executed.

          This might be caused by the fact that all fault injection tests are executed (as well as their results are collected) under build-fi/, which differs from the standard location.

          Show
          Konstantin Boudnik added a comment - Note that the fault inject tests do not show up in the test result page although they have been executed. This might be caused by the fact that all fault injection tests are executed (as well as their results are collected) under build-fi/ , which differs from the standard location.
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed]
          Tsz Wo Nicholas Sze made changes -
          Attachment h451_20090807.patch [ 12415871 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h451_20090807.patch: removed testPipelineFi12

          Show
          Tsz Wo Nicholas Sze added a comment - h451_20090807.patch: removed testPipelineFi12
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The failed tests, TestBlockReplacement and TestNameNodeMetrics, are not related to this. This issue adds new fault inject tests but does not change existing codes except that the patch adds a few getter methods.

          Unfortunately, Hudson fails on testPipelineFi12.

          ...
               [exec]     [junit] Tests run: 5, Failures: 0, Errors: 1, Time elapsed: 52.005 sec
               [exec]     [junit] 2009-08-07 02:27:44,904 ERROR hdfs.DFSClient (DFSClient.java:close(1084))
           - Exception closing file /testPipelineFi12/foo : java.io.IOException: Could not get block locations. Source file "/testPipelineFi12/foo" - Aborting...
               [exec]     [junit] java.io.IOException: Could not get block locations. Source file "/testPipelineFi12/foo" - Aborting...
               [exec]     [junit] 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.processDatanodeError(DFSClient.java:2642)
               [exec]     [junit] 	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2366)
               [exec]     [junit] Test org.apache.hadoop.hdfs.server.datanode.TestFiDataTransferProtocol FAILED
               [exec] 
               [exec] checkfailure:
               [exec]     [touch] Creating /home/hudson/hudson-slave/workspace/Hdfs-Patch-vesta.apache.org/trunk/build-fi/test/testsfailed
          

          Note that the fault inject tests do not show up in the test result page although they have been executed.

          Show
          Tsz Wo Nicholas Sze added a comment - The failed tests, TestBlockReplacement and TestNameNodeMetrics, are not related to this. This issue adds new fault inject tests but does not change existing codes except that the patch adds a few getter methods. Unfortunately, Hudson fails on testPipelineFi12. ... [exec] [junit] Tests run: 5, Failures: 0, Errors: 1, Time elapsed: 52.005 sec [exec] [junit] 2009-08-07 02:27:44,904 ERROR hdfs.DFSClient (DFSClient.java:close(1084)) - Exception closing file /testPipelineFi12/foo : java.io.IOException: Could not get block locations. Source file "/testPipelineFi12/foo" - Aborting... [exec] [junit] java.io.IOException: Could not get block locations. Source file "/testPipelineFi12/foo" - Aborting... [exec] [junit] at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.processDatanodeError(DFSClient.java:2642) [exec] [junit] at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2366) [exec] [junit] Test org.apache.hadoop.hdfs.server.datanode.TestFiDataTransferProtocol FAILED [exec] [exec] checkfailure: [exec] [touch] Creating /home/hudson/hudson-slave/workspace/Hdfs-Patch-vesta.apache.org/trunk/build-fi/test/testsfailed Note that the fault inject tests do not show up in the test result page although they have been executed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12415785/h451_20090806.patch
          against trunk revision 801736.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415785/h451_20090806.patch against trunk revision 801736. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/47/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks for the quick review, Cos.

          BTW, are you going to comment on Kan's patch in HADOOP-6176? They are asking me for committing it.

          Nicholas

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks for the quick review, Cos. BTW, are you going to comment on Kan's patch in HADOOP-6176 ? They are asking me for committing it. Nicholas
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good

          Show
          Konstantin Boudnik added a comment - +1 patch looks good
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
          Fix Version/s 0.21.0 [ 12314046 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Submitting the first set of fault injection tests to Hudson...

          Show
          Tsz Wo Nicholas Sze added a comment - Submitting the first set of fault injection tests to Hudson...
          Tsz Wo Nicholas Sze made changes -
          Attachment h451_20090806.patch [ 12415785 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h451_20090806.patch: Includes unit tests Pipeline_Fi_06,07,12,14,15 described in the AppendTestPlan.

          Show
          Tsz Wo Nicholas Sze added a comment - h451_20090806.patch: Includes unit tests Pipeline_Fi_06,07,12,14,15 described in the AppendTestPlan .
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-483 [ HDFS-483 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-521 [ HDFS-521 ]
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-265 [ HDFS-265 ]
          Tsz Wo Nicholas Sze created issue -

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development