Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1853

MultipleOutputs does not cache TaskAttemptContext

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.21.0, 0.22.0
    • Component/s: task
    • Labels:
      None
    • Environment:

      OSX 10.6
      java6

    • Hadoop Flags:
      Reviewed

      Description

      In MultipleOutputs there is

       private TaskAttemptContext getContext(String nameOutput) throws IOException {
          // The following trick leverages the instantiation of a record writer via
          // the job thus supporting arbitrary output formats.
          Job job = new Job(context.getConfiguration());
          job.setOutputFormatClass(getNamedOutputFormatClass(context, nameOutput));
          job.setOutputKeyClass(getNamedOutputKeyClass(context, nameOutput));
          job.setOutputValueClass(getNamedOutputValueClass(context, nameOutput));
          TaskAttemptContext taskContext = 
            new TaskAttemptContextImpl(job.getConfiguration(), 
                                       context.getTaskAttemptID());
          return taskContext;
        }
      

      so for every reduce call it creates a new Job instance ...which creates a new LocalJobRunner.
      That does not sound like a good idea.

      You end up with a flood of "jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized"

      This should probably also be added to 0.22.

        Issue Links

          Activity

          Hide
          Amareshwari Sriramadasu added a comment -

          so for every reduce call it creates a new Job instance ...which creates a new LocalJobRunner.

          Though this will be fixed by MAPREDUCE-1505, caching introduced in the patch is good optimization.

          Can you regenerate the patch with --no-prefix option, so that it can be applied with command "patch -p0 < patch-file" ?

          Show
          Amareshwari Sriramadasu added a comment - so for every reduce call it creates a new Job instance ...which creates a new LocalJobRunner. Though this will be fixed by MAPREDUCE-1505 , caching introduced in the patch is good optimization. Can you regenerate the patch with --no-prefix option, so that it can be applied with command "patch -p0 < patch-file" ?
          Hide
          Torsten Curdt added a comment -

          with --no-prefix

          Show
          Torsten Curdt added a comment - with --no-prefix
          Hide
          Torsten Curdt added a comment -

          Patch updated. Should apply also with -p0 now.

          Show
          Torsten Curdt added a comment - Patch updated. Should apply also with -p0 now.
          Hide
          Amareshwari Sriramadasu added a comment -

          Changes look good.

          Show
          Amareshwari Sriramadasu added a comment - Changes look good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446976/cache-task-attempts.diff
          against trunk revision 953976.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446976/cache-task-attempts.diff against trunk revision 953976. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/241/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Test failure is due to MAPREDUCE-1834

          Show
          Amareshwari Sriramadasu added a comment - Test failure is due to MAPREDUCE-1834
          Hide
          Torsten Curdt added a comment -

          Anything else I can do here?

          Would also like to see this included into 0.21 (not just 0.22)

          Show
          Torsten Curdt added a comment - Anything else I can do here? Would also like to see this included into 0.21 (not just 0.22)
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch cleanly applies to 0.21 as well. Will commit it if nobody objects.

          Show
          Amareshwari Sriramadasu added a comment - Patch cleanly applies to 0.21 as well. Will commit it if nobody objects.
          Hide
          Amareshwari Sriramadasu added a comment -

          I just committed this to trunk and branch 0.21. Thanks Torsen !

          Show
          Amareshwari Sriramadasu added a comment - I just committed this to trunk and branch 0.21. Thanks Torsen !
          Hide
          Amareshwari Sriramadasu added a comment -

          I just committed this to trunk and branch 0.21. Thanks Torsen !

          Sorry for the misspelled name. I meant Thanks Torsten !

          Show
          Amareshwari Sriramadasu added a comment - I just committed this to trunk and branch 0.21. Thanks Torsen ! Sorry for the misspelled name. I meant Thanks Torsten !
          Hide
          David Rosenstrauch added a comment -

          Is there any workaround to this issue for those of us who are still running 0.20?

          I have a job that very much lends itself to using the MultipleOutputs functionality, but I can't use it if a bug like this crushes the job's performance.

          Anyone have any alternate suggestions?

          Show
          David Rosenstrauch added a comment - Is there any workaround to this issue for those of us who are still running 0.20? I have a job that very much lends itself to using the MultipleOutputs functionality, but I can't use it if a bug like this crushes the job's performance. Anyone have any alternate suggestions?

            People

            • Assignee:
              Torsten Curdt
              Reporter:
              Torsten Curdt
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development