Hadoop Common
  1. Hadoop Common
  2. HADOOP-7298

Add test utility for writing multi-threaded tests

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.

      In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

      1. hadoop-7298.txt
        9 kB
        Todd Lipcon
      2. hadoop-7298.txt
        12 kB
        Harsh J
      3. hadoop-7298.txt
        12 kB
        Harsh J

        Issue Links

          Activity

          Show
          Todd Lipcon added a comment - The HBase class lives here: http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/MultithreadedTestUtil.java For an example usage, see testLABThreading in the following test: http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
          Hide
          Todd Lipcon added a comment -

          Cleaned up the hbase utility a bit and added test cases for the test utility.

          I think this will make a nice building block to build some stress/fuzz tests for the namenode for HDFS-988. But, no need to commit it until such a test is ready.

          Show
          Todd Lipcon added a comment - Cleaned up the hbase utility a bit and added test cases for the test utility. I think this will make a nice building block to build some stress/fuzz tests for the namenode for HDFS-988 . But, no need to commit it until such a test is ready.
          Hide
          Suresh Srinivas added a comment -

          Todd, given that this is a utility based on which tests are going to be written, can you add javadoc and possibly examples or brief description on how to use it.

          Show
          Suresh Srinivas added a comment - Todd, given that this is a utility based on which tests are going to be written, can you add javadoc and possibly examples or brief description on how to use it.
          Hide
          Harsh J added a comment -

          Ditto code patch attached, with following changes:

          +JavaDocs
          -Whitespaces

          Show
          Harsh J added a comment - Ditto code patch attached, with following changes: +JavaDocs -Whitespaces
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483809/hadoop-7298.txt
          against trunk revision 1139476.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483809/hadoop-7298.txt against trunk revision 1139476. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Looks good to me. Suresh, mind looking at this latest patch? Harsh's patch for MAPREDUCE-1347 depends on this.

          Show
          Todd Lipcon added a comment - Looks good to me. Suresh, mind looking at this latest patch? Harsh's patch for MAPREDUCE-1347 depends on this.
          Hide
          Suresh Srinivas added a comment -

          +1

          Show
          Suresh Srinivas added a comment - +1
          Hide
          Konstantin Boudnik added a comment -

          Todd, won't class TestThread be picked up by ant junit task? I'd suggest to change it ThreadTest to avoid it.

          Show
          Konstantin Boudnik added a comment - Todd, won't class TestThread be picked up by ant junit task? I'd suggest to change it ThreadTest to avoid it.
          Hide
          Harsh J added a comment -

          Refactored TestThread to TestingThread. Updating the MAPREDUCE-1347 patch accordingly.

          Show
          Harsh J added a comment - Refactored TestThread to TestingThread . Updating the MAPREDUCE-1347 patch accordingly.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487006/hadoop-7298.txt
          against trunk revision 1147971.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/746//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/746//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/746//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487006/hadoop-7298.txt against trunk revision 1147971. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/746//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/746//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/746//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          If there are no more comments, and things look good in this patch, then can this go in so MAPREDUCE-1347 may be re-reviewed+resolved as well?

          Thanks for all the reviews!

          Show
          Harsh J added a comment - If there are no more comments, and things look good in this patch, then can this go in so MAPREDUCE-1347 may be re-reviewed+resolved as well? Thanks for all the reviews!
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and 22. (22 since it's only test code and necessary for the bug fix MAPREDUCE-1347)

          Show
          Todd Lipcon added a comment - Committed to trunk and 22. (22 since it's only test code and necessary for the bug fix MAPREDUCE-1347 )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #698 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/698/)
          HADOOP-7298. Add test utility for writing multi-threaded tests. Contributed by Todd Lipcon and Harsh J Chouraria.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150910
          Files :

          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/MultithreadedTestUtil.java
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/TestMultithreadedTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #698 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/698/ ) HADOOP-7298 . Add test utility for writing multi-threaded tests. Contributed by Todd Lipcon and Harsh J Chouraria. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150910 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/MultithreadedTestUtil.java /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/TestMultithreadedTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #66 (See https://builds.apache.org/job/Hadoop-Common-22-branch/66/)
          HADOOP-7298. Add test utility for writing multi-threaded tests. Contributed by Todd Lipcon and Harsh J Chouraria.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150912
          Files :

          • /hadoop/common/branches/branch-0.22/common/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/test/MultithreadedTestUtil.java
          • /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/test/TestMultithreadedTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #66 (See https://builds.apache.org/job/Hadoop-Common-22-branch/66/ ) HADOOP-7298 . Add test utility for writing multi-threaded tests. Contributed by Todd Lipcon and Harsh J Chouraria. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150912 Files : /hadoop/common/branches/branch-0.22/common/CHANGES.txt /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/test/MultithreadedTestUtil.java /hadoop/common/branches/branch-0.22/common/src/test/core/org/apache/hadoop/test/TestMultithreadedTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #758 (See https://builds.apache.org/job/Hadoop-Common-trunk/758/)
          HADOOP-7298. Add test utility for writing multi-threaded tests. Contributed by Todd Lipcon and Harsh J Chouraria.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150910
          Files :

          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/MultithreadedTestUtil.java
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/TestMultithreadedTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #758 (See https://builds.apache.org/job/Hadoop-Common-trunk/758/ ) HADOOP-7298 . Add test utility for writing multi-threaded tests. Contributed by Todd Lipcon and Harsh J Chouraria. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150910 Files : /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/MultithreadedTestUtil.java /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/test/TestMultithreadedTestUtil.java
          Hide
          Joep Rottinghuis added a comment -

          Todd, in the waitFor method why do you keep a count of finished and then wait in a loop?
          Wouldn't a CountDownLatch work at least as well?

          A problem I'm seeing in many tests is that they sleep for some # seconds in a loop.
          For one single test that does not seem to matter too much, but adding it all up that would result in a lot of unnecessary waiting. If one waits x seconds in a loop, then on average 1/2 x of time is wasted. The total quite of tests runs in about 4 hours (on 0.20-security) even when I throw bigger hardware at it.
          When a CountDownLatch is used to coordinate between threads (either to signal that work can start, or that threads have finished) then there is no such additional waiting.

          Show
          Joep Rottinghuis added a comment - Todd, in the waitFor method why do you keep a count of finished and then wait in a loop? Wouldn't a CountDownLatch work at least as well? A problem I'm seeing in many tests is that they sleep for some # seconds in a loop. For one single test that does not seem to matter too much, but adding it all up that would result in a lot of unnecessary waiting. If one waits x seconds in a loop, then on average 1/2 x of time is wasted. The total quite of tests runs in about 4 hours (on 0.20-security) even when I throw bigger hardware at it. When a CountDownLatch is used to coordinate between threads (either to signal that work can start, or that threads have finished) then there is no such additional waiting.
          Hide
          Todd Lipcon added a comment -

          When a thread finishes, it calls notify(). So, if the main thread is waiting, it will be woken up.

          Show
          Todd Lipcon added a comment - When a thread finishes, it calls notify(). So, if the main thread is waiting, it will be woken up.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development