Hadoop Common
  1. Hadoop Common
  2. HADOOP-5621

MapReducer to run junit tests under Hadoop

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.21.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This is something I mentioned to some people last week, thought I would start a discussion on it.

      We could run junit tests as a MapReduce job with

      1. a mapper that takes a list of classes, one per line
      2. extracts the test suite from each class, and then invokes each test method. This would be a new junit test runner.
      3. saves the result (and any exceptions) as the output. Also saves any machine specific details.
      4. It also needs to grab the System.out and System.err channels, to map them to specific tests.
      5. Measure how long the tests took (incuding setup/teardown time)
      6. Add an ant task <listresources> to take filesets and other patterns, and generate text files from the contents (with stripping of prefixes and suffices, directory separator substition, file begin/end values, etc, etc). I have this with tests already.

      The result would be that you could point listresources at a directory tree and create a text file listing all tests to run. These could be executed across multiple hosts and the results correlated. It would be, initially, a MapExpand, as the output would be bigger than the input

      Feature creep then becomes the analysis

      1. Add another MR class which runs through all failing tests and creates a new list of test classes that failed. This could be rescheduled on different runs, and makes for a faster cycle (only run failing tests until they work)
      2. Add something to only get failing tests, summarise them (somehow) in a user readable form
      3. Something to get partially failing tests and highlight machine differences.
      4. Add something to compare tests over time, detect those which are getting slower?
      5. an MR to regenerate the classic Ant junit XML reports, for presentation in other tools (like hudson)

        Issue Links

          Activity

          Hide
          Steve Loughran added a comment -

          YARN lets us run tests across a cluster differently

          Show
          Steve Loughran added a comment - YARN lets us run tests across a cluster differently
          Hide
          steve_l added a comment -

          HADOOP-1527 is running Ant's <junit> task as the job, runs the tests in a different VM, generating, text output which it then tries to parse in. It will need ant.jar, ant-optional.jar and junit on the path.

          This one does it in VM -which has a price -less robust against JVM crashes, and there isn't any timeout enforcement logic there yet. Where the big difference is is that the output comes out serialized, one test result is emitted for every method that passes/fails, with the exceptions and stack trace included. This makes it a more powerful format for analysis.

          The brittleness/timeout is a problem, but then its a problem in <junit> too. What could be done would be to exec() a child process for doing the work, with it throwing back the serialized test result, with the MR host process killing things if they took too long -but still saving everything that got forwarded out. That would be a trickier undertaking, but possible.

          Also: I pass down a configuration to the test cases if they have the interface to receive it. Right now, that is the same config that the MR job gets, but you could imagine more configuration-driven work where every map line includes not just the test method but some config information, so someone could do more configuration driven work. Of course, if you want to do that then the test reports had better include the entire configuration, for better examination of what triggered the failures.

          It's currently built against junit3.8.2; moving to junit4 is worthwhile, but I'd have to use a slightly different test runner

          What '1257 has is an ant task that looks more like a drop in replacement for <junit>; a specific launcher for the job. This proposals workflow would start off generating a list of tests to run then pushing that out as part of a blocking job submission.

          Show
          steve_l added a comment - HADOOP-1527 is running Ant's <junit> task as the job, runs the tests in a different VM, generating, text output which it then tries to parse in. It will need ant.jar, ant-optional.jar and junit on the path. This one does it in VM -which has a price -less robust against JVM crashes, and there isn't any timeout enforcement logic there yet. Where the big difference is is that the output comes out serialized, one test result is emitted for every method that passes/fails, with the exceptions and stack trace included. This makes it a more powerful format for analysis. The brittleness/timeout is a problem, but then its a problem in <junit> too. What could be done would be to exec() a child process for doing the work, with it throwing back the serialized test result, with the MR host process killing things if they took too long -but still saving everything that got forwarded out. That would be a trickier undertaking, but possible. Also: I pass down a configuration to the test cases if they have the interface to receive it. Right now, that is the same config that the MR job gets, but you could imagine more configuration-driven work where every map line includes not just the test method but some config information, so someone could do more configuration driven work. Of course, if you want to do that then the test reports had better include the entire configuration, for better examination of what triggered the failures. It's currently built against junit3.8.2; moving to junit4 is worthwhile, but I'd have to use a slightly different test runner What '1257 has is an ant task that looks more like a drop in replacement for <junit>; a specific launcher for the job. This proposals workflow would start off generating a list of tests to run then pushing that out as part of a blocking job submission.
          Hide
          Nigel Daley added a comment -

          Steve, how's this different than HADOOP-1257?

          Show
          Nigel Daley added a comment - Steve, how's this different than HADOOP-1257 ?
          Hide
          steve_l added a comment -

          I should add that my prototype work is currently in a different SCM repo, still LGPL, but moving to Apache licensed shortly.

          http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/junitmr/

          One thing I also do in this code is pass down a configuration to every test case. This will give the hadoop-only tests more information. It would make them only work in this test runner though.

          if (testSuite instanceof JUnitHadoopContext) {
              JUnitHadoopContext ctx = (JUnitHadoopContext) testSuite;
              ctx.setConfiguration(context.getConfiguration());
          }
          

          The listresources task, to create a text file listing all tests in a source tree, is elsewhere
          http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/extras/ant/src/org/smartfrog/tools/ant/

          this stuff is just a start

          1. need to capture the IO streams
          2. need the post-processing
          3. the test runner probably needs to be more robust
          4. I'd like to add a SkippedException so that when a test case chooses not to run, it throws that and the outcome is logged differently.
          Show
          steve_l added a comment - I should add that my prototype work is currently in a different SCM repo, still LGPL, but moving to Apache licensed shortly. http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/junitmr/ One thing I also do in this code is pass down a configuration to every test case. This will give the hadoop-only tests more information. It would make them only work in this test runner though. if (testSuite instanceof JUnitHadoopContext) { JUnitHadoopContext ctx = (JUnitHadoopContext) testSuite; ctx.setConfiguration(context.getConfiguration()); } The listresources task, to create a text file listing all tests in a source tree, is elsewhere http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/extras/ant/src/org/smartfrog/tools/ant/ this stuff is just a start need to capture the IO streams need the post-processing the test runner probably needs to be more robust I'd like to add a SkippedException so that when a test case chooses not to run, it throws that and the outcome is logged differently.

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development