Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2350

LocalJobRunner uses "mapred.output.committer.class" configuration property to retrieve the OutputCommitter (regardless of whether the old API is used or the new API)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.

      See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();

      Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.

      1. MAPREDUCE-2350-1.patch
        8 kB
        Devaraj K
      2. MAPREDUCE-2350.patch
        8 kB
        Devaraj K

        Issue Links

          Activity

          Anthony Hsu made changes -
          Summary LocalJobRunner uses "mapred.output.committer.class" configuration property to retreive the OutputCommitter (regardless of whether the old API is used or the new API) LocalJobRunner uses "mapred.output.committer.class" configuration property to retrieve the OutputCommitter (regardless of whether the old API is used or the new API)
          Anthony Hsu made changes -
          Summary LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API) LocalJobRunner uses "mapred.output.committer.class" configuration property to retreive the OutputCommitter (regardless of whether the old API is used or the new API)
          Devaraj K made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.24.0 [ 12317654 ]
          Resolution Duplicate [ 3 ]
          Devaraj K made changes -
          Link This issue is duplicated by MAPREDUCE-3563 [ MAPREDUCE-3563 ]
          Arun C Murthy made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Devaraj K made changes -
          Attachment MAPREDUCE-2350-1.patch [ 12493562 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Devaraj K [ devaraj.k ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Devaraj K made changes -
          Attachment MAPREDUCE-2350.patch [ 12489188 ]
          Ahmed Radwan made changes -
          Field Original Value New Value
          Summary LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of wether the old API is used or the new API) LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
          Ahmed Radwan created issue -

            People

            • Assignee:
              Devaraj K
              Reporter:
              Ahmed Radwan
            • Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development