Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1367

LocalJobRunner should support parallel mapper execution

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs.

      1. MAPREDUCE-1367.patch
        22 kB
        Aaron Kimball
      2. MAPREDUCE-1367.2.patch
        24 kB
        Aaron Kimball
      3. MAPREDUCE-1367.3.patch
        24 kB
        Aaron Kimball
      4. MAPREDUCE-1367.4.patch
        26 kB
        Aaron Kimball
      5. MAPREDUCE-1367.5.patch
        28 kB
        Aaron Kimball
      6. MAPREDUCE-1367.6.patch
        28 kB
        Aaron Kimball
      7. MAPREDUCE-1367.7.patch
        29 kB
        Aaron Kimball
      8. MAPREDUCE-1367-branch1.patch
        28 kB
        Sandy Ryza

        Issue Links

          Activity

          Aaron Kimball created issue -
          Hide
          Aaron Kimball added a comment -

          Attaching a patch that implements this improvement. This patch includes a test case which launches 6 mappers concurrently; these mappers run on a variety of schedules (some are faster, some are slower) in an attempt to suss out any race conditions that might develop.

          The level of parallelism is controlled by a new parameter: mapred.local.map.tasks.maximum. This defaults to 1, so that unspecified behavior is as before.

          I also tested this by running the 'pi' example from the command line:

          bin/hadoop jar hadoop-mapred-examples-0.22.0-SNAPSHOT.jar pi -D mapreduce.jobtracker.address=local -D mapreduce.local.map.tasks.maximum=2 20 5000000
          

          With mapreduce.local.map.tasks.maximum set to 1, this takes 13.5 seconds on my machine. With it set to 2 or above (I have two cores), the runtime drops to 8.5 seconds.

          Show
          Aaron Kimball added a comment - Attaching a patch that implements this improvement. This patch includes a test case which launches 6 mappers concurrently; these mappers run on a variety of schedules (some are faster, some are slower) in an attempt to suss out any race conditions that might develop. The level of parallelism is controlled by a new parameter: mapred.local.map.tasks.maximum . This defaults to 1, so that unspecified behavior is as before. I also tested this by running the 'pi' example from the command line: bin/hadoop jar hadoop-mapred-examples-0.22.0-SNAPSHOT.jar pi -D mapreduce.jobtracker.address=local -D mapreduce.local.map.tasks.maximum=2 20 5000000 With mapreduce.local.map.tasks.maximum set to 1, this takes 13.5 seconds on my machine. With it set to 2 or above (I have two cores), the runtime drops to 8.5 seconds.
          Aaron Kimball made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-1367.patch [ 12429800 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429800/MAPREDUCE-1367.patch
          against trunk revision 897118.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429800/MAPREDUCE-1367.patch against trunk revision 897118. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/369/console This message is automatically generated.
          Aaron Kimball made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Aaron Kimball added a comment -

          New patch; rewrites counter handling to make TestJobCounters pass.

          Show
          Aaron Kimball added a comment - New patch; rewrites counter handling to make TestJobCounters pass.
          Aaron Kimball made changes -
          Attachment MAPREDUCE-1367.2.patch [ 12429958 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429958/MAPREDUCE-1367.2.patch
          against trunk revision 898019.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429958/MAPREDUCE-1367.2.patch against trunk revision 898019. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/261/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          Tests that failed in Hudson all pass locally. These tests don't appear to use the LocalJobRunner.

          Show
          Aaron Kimball added a comment - Tests that failed in Hudson all pass locally. These tests don't appear to use the LocalJobRunner.
          Hide
          Aaron Kimball added a comment -

          new patch addresses findbugs warnings.

          Show
          Aaron Kimball added a comment - new patch addresses findbugs warnings.
          Aaron Kimball made changes -
          Attachment MAPREDUCE-1367.3.patch [ 12429995 ]
          Aaron Kimball made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Chris Douglas added a comment -

          As a first pass:

          • map_tasks is volatile, but increment/decrement is not atomic. Consider AtomicInteger instead
          • getMapExecutor should be named createMapExecutor ; adding tasks to the ExecutorService belongs at the caller. The logging should probably be at DEBUG instead of INFO
          • storedException should probably be Throwable instead of Exception
          • if LOCAL_MAX_MAPS is < 1, that should be an error

          Otherwise, this looks pretty good.

          Show
          Chris Douglas added a comment - As a first pass: map_tasks is volatile, but increment/decrement is not atomic. Consider AtomicInteger instead getMapExecutor should be named createMapExecutor ; adding tasks to the ExecutorService belongs at the caller. The logging should probably be at DEBUG instead of INFO storedException should probably be Throwable instead of Exception if LOCAL_MAX_MAPS is < 1, that should be an error Otherwise, this looks pretty good.
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Aaron Kimball added a comment -

          Chris, all those suggestions make sense. Here's a new patch that implements them.

          I also added a static getter/setter API for the config param

          public static void LocalJobRunner.setLocalMaxRunningMaps(JobContext, int)
          public static int LocalJobRunner.getLocalMaxRunningMaps(JobContext)
          
          Show
          Aaron Kimball added a comment - Chris, all those suggestions make sense. Here's a new patch that implements them. I also added a static getter/setter API for the config param public static void LocalJobRunner.setLocalMaxRunningMaps(JobContext, int ) public static int LocalJobRunner.getLocalMaxRunningMaps(JobContext)
          Aaron Kimball made changes -
          Attachment MAPREDUCE-1367.4.patch [ 12430043 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Todd Lipcon added a comment -
          • MapTaskRunnable: make most of the instance variables final?
          • it should be clearer that mapOutputFiles is an "out param" for getMapTaskRunnables - ie that it's expected to be intitially empty. Alternatively, it looks like you may be able to remove this variable entirely by instead storing the MapOutputFile in the MapRunnable instance, and then iterating directly over the MapRunnables in the reducer. Does that make sense? However, the way you've done it is less invasive to the reduce side, so if you don't see the benefit, feel free to ignore this suggestion.
          • Does this handle the degenerate case of 0-map jobs? It sounds ridiculous, but I recall previous JIRAs for this situation, since occasionally people have a cron job that periodically processes a given directory. If the directory is empty, it may generate a job with no input splits and thus no tasks.

          Aside from that, looks good to me.

          Show
          Todd Lipcon added a comment - MapTaskRunnable: make most of the instance variables final? it should be clearer that mapOutputFiles is an "out param" for getMapTaskRunnables - ie that it's expected to be intitially empty. Alternatively, it looks like you may be able to remove this variable entirely by instead storing the MapOutputFile in the MapRunnable instance, and then iterating directly over the MapRunnables in the reducer. Does that make sense? However, the way you've done it is less invasive to the reduce side, so if you don't see the benefit, feel free to ignore this suggestion. Does this handle the degenerate case of 0-map jobs? It sounds ridiculous, but I recall previous JIRAs for this situation, since occasionally people have a cron job that periodically processes a given directory. If the directory is empty, it may generate a job with no input splits and thus no tasks. Aside from that, looks good to me.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12430043/MAPREDUCE-1367.4.patch
          against trunk revision 898486.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430043/MAPREDUCE-1367.4.patch against trunk revision 898486. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/378/console This message is automatically generated.
          Aaron Kimball made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Aaron Kimball added a comment -

          Todd, good catch: it didn't handle the 0-mapper case. It does now. Testcase included.

          Show
          Aaron Kimball added a comment - Todd, good catch: it didn't handle the 0-mapper case. It does now. Testcase included.
          Aaron Kimball made changes -
          Attachment MAPREDUCE-1367.5.patch [ 12430081 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12430081/MAPREDUCE-1367.5.patch
          against trunk revision 898486.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430081/MAPREDUCE-1367.5.patch against trunk revision 898486. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/268/console This message is automatically generated.
          Aaron Kimball made changes -
          Link This issue blocks MAPREDUCE-434 [ MAPREDUCE-434 ]
          Hide
          Jeff Hammerbacher added a comment -

          What is the delay in committing this patch?

          Show
          Jeff Hammerbacher added a comment - What is the delay in committing this patch?
          Hide
          Tom White added a comment -

          Aaron, this patch has gone stale. Could you regenerate, then I think it's ready to be committed.

          Show
          Tom White added a comment - Aaron, this patch has gone stale. Could you regenerate, then I think it's ready to be committed.
          Hide
          Aaron Kimball added a comment -

          new patch, sync'd with trunk.

          Show
          Aaron Kimball added a comment - new patch, sync'd with trunk.
          Aaron Kimball made changes -
          Attachment MAPREDUCE-1367.6.patch [ 12431509 ]
          Aaron Kimball made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Aaron Kimball [ kimballa ] ZhuGuanyin [ buptzhugy ]
          Aaron Kimball made changes -
          Assignee ZhuGuanyin [ buptzhugy ] Aaron Kimball [ kimballa ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431509/MAPREDUCE-1367.6.patch
          against trunk revision 903508.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431509/MAPREDUCE-1367.6.patch against trunk revision 903508. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/288/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          Test failures are all unrelated (TT blacklisting has now falied two tests in a row?)

          Show
          Aaron Kimball added a comment - Test failures are all unrelated (TT blacklisting has now falied two tests in a row?)
          Hide
          Aaron Kimball added a comment -

          new patch, sync'd with trunk.

          Show
          Aaron Kimball added a comment - new patch, sync'd with trunk.
          Aaron Kimball made changes -
          Attachment MAPREDUCE-1367.7.patch [ 12431823 ]
          Aaron Kimball made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431823/MAPREDUCE-1367.7.patch
          against trunk revision 904609.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431823/MAPREDUCE-1367.7.patch against trunk revision 904609. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/418/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          Test failures seem unrelated - classpath issues on hudson again?

          Show
          Aaron Kimball added a comment - Test failures seem unrelated - classpath issues on hudson again?
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Aaron!

          Show
          Tom White added a comment - I've just committed this. Thanks Aaron!
          Tom White made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314184 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/)
          . LocalJobRunner should support parallel mapper execution. Contributed by Aaron Kimball.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #221 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/221/ ) . LocalJobRunner should support parallel mapper execution. Contributed by Aaron Kimball.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
          Tom White made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Ramkumar Vadali added a comment -

          @Aaron, Just curious, is this being used in production? If so, could you please outline the use case?

          Show
          Ramkumar Vadali added a comment - @Aaron, Just curious, is this being used in production? If so, could you please outline the use case?
          Forest Tan made changes -
          Link This issue is cloned as MAPREDUCE-2931 [ MAPREDUCE-2931 ]
          Sandy Ryza made changes -
          Attachment MAPREDUCE-1367-branch1.patch [ 12548014 ]
          Hide
          Sandy Ryza added a comment -

          I submitted a patch for branch-1, as it was never backported.

          Show
          Sandy Ryza added a comment - I submitted a patch for branch-1, as it was never backported.
          Hide
          Sandy Ryza added a comment -

          Regenerated branch-1 patch

          Show
          Sandy Ryza added a comment - Regenerated branch-1 patch
          Hide
          Sandy Ryza added a comment -

          TestLocalRunner passes

          Show
          Sandy Ryza added a comment - TestLocalRunner passes
          Gavin made changes -
          Link This issue blocks MAPREDUCE-434 [ MAPREDUCE-434 ]
          Gavin made changes -
          Link This issue is depended upon by MAPREDUCE-434 [ MAPREDUCE-434 ]
          Hide
          Johannes Zillmann added a comment -

          Hey guys, is 0.21.0 the only branch this is fixed in ?
          Not in 1.x, not in 2.x ?

          Show
          Johannes Zillmann added a comment - Hey guys, is 0.21.0 the only branch this is fixed in ? Not in 1.x, not in 2.x ?
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          18d 7h 29m 6 Aaron Kimball 29/Jan/10 21:47
          Open Open Patch Available Patch Available
          2d 14h 43m 7 Aaron Kimball 29/Jan/10 21:47
          Patch Available Patch Available Resolved Resolved
          3h 36m 1 Tom White 30/Jan/10 01:24
          Resolved Resolved Closed Closed
          206d 19h 55m 1 Tom White 24/Aug/10 21:19

            People

            • Assignee:
              Aaron Kimball
              Reporter:
              Aaron Kimball
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development