Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4280

LocalJobRunner doesn't honor custom OutputCommiter cleanupJob

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      I have an implementation SampleFileOutputCommiter which extends org.apache.hadoop.mapred.FileOutputCommitter . The implementation has specific code to be executed during cleanupJob() execution.
      When the framework(LocalJobRunner) makes a call to commitJob(), the framework never takes care of calling the cleanupJob() instead it calls org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitJob() which internally calls its own cleanupJob().
      Though the method cleanupJob is deprecated but, still I feel the framework should take care of executing it as it is being executed from org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.
      Currently the framework is not letting the Jobs written with MRV1 to run properly.

        Issue Links

          Activity

          Hide
          qwertymaniac Harsh J added a comment -

          This is fixed in MR2's LocalJobRunner in 2.x and trunk via MAPREDUCE-3563. We probably have to backport it to 1.x.

          Show
          qwertymaniac Harsh J added a comment - This is fixed in MR2's LocalJobRunner in 2.x and trunk via MAPREDUCE-3563 . We probably have to backport it to 1.x.
          Hide
          qwertymaniac Harsh J added a comment -

          Changing affected versions.

          Show
          qwertymaniac Harsh J added a comment - Changing affected versions.
          Hide
          subrotosanyal Subroto Sanyal added a comment -

          This is fixed in MR2's LocalJobRunner in 2.x and trunk via MAPREDUCE-3563. We probably have to backport it to 1.x.

          I feel even this fix doesn't honors the cleanupJob of CustomCommitter. Seems needs to be fixed in some 2.x as well.

          Show
          subrotosanyal Subroto Sanyal added a comment - This is fixed in MR2's LocalJobRunner in 2.x and trunk via MAPREDUCE-3563 . We probably have to backport it to 1.x. I feel even this fix doesn't honors the cleanupJob of CustomCommitter. Seems needs to be fixed in some 2.x as well.
          Hide
          qwertymaniac Harsh J added a comment -

          Ok you've discovered a second, valid issue. Can we repurpose this JIRA to fix that, or would you like to see this backported to 1.x-based releases too? In the latter case, please file a second JIRA to fix and add a cleanup call.

          Show
          qwertymaniac Harsh J added a comment - Ok you've discovered a second, valid issue. Can we repurpose this JIRA to fix that, or would you like to see this backported to 1.x-based releases too? In the latter case, please file a second JIRA to fix and add a cleanup call.
          Hide
          subrotosanyal Subroto Sanyal added a comment -

          I feel the current JIRA description targets the "second, valid issue".
          I think no need to re-purpose the JIRA (may be we need to change the Affects Version).

          would you like to see this backported to 1.x-based releases too?

          Raised a new one MAPREDUCE-4281

          Show
          subrotosanyal Subroto Sanyal added a comment - I feel the current JIRA description targets the "second, valid issue" . I think no need to re-purpose the JIRA (may be we need to change the Affects Version). would you like to see this backported to 1.x-based releases too? Raised a new one MAPREDUCE-4281
          Hide
          mattf Matt Foley added a comment -

          Moved to 1.2.0 upon release of 1.1.0.

          Show
          mattf Matt Foley added a comment - Moved to 1.2.0 upon release of 1.1.0.
          Hide
          mattf Matt Foley added a comment -

          Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

          Show
          mattf Matt Foley added a comment - Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

            People

            • Assignee:
              Unassigned
              Reporter:
              subrotosanyal Subroto Sanyal
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development