Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6009

Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.2.1
    • Fix Version/s: 1.3.0, 1.2.2
    • Component/s: client, job submission
    • Labels:
      None

      Description

      In branch 1 job commit is executed in a JOB_CLEANUP task that may run in either map or reduce slot

      in org.apache.hadoop.mapreduce.Job#setUseNewAPI there is a logic setting new-api flag only for reduce-ful jobs.

          if (numReduces != 0) {
            conf.setBooleanIfUnset("mapred.reducer.new-api",
                                   conf.get(oldReduceClass) == null);
            ...
      

      Therefore, when cleanup runs in a reduce slot, ReduceTask inits using the old API and runs incorrect default OutputCommitter, instead of consulting OutputFormat.

        Activity

        Sandy Ryza made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 1.3.0 [ 12324153 ]
        Fix Version/s 1.2.2 [ 12324861 ]
        Resolution Fixed [ 1 ]
        Gera Shegalov made changes -
        Attachment MAPREDUCE-6009.v02-branch-1.2.patch [ 12674127 ]
        Gera Shegalov made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Gera Shegalov made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-6009.v01-branch-1.2.patch [ 12658096 ]
        Gera Shegalov created issue -

          People

          • Assignee:
            Gera Shegalov
            Reporter:
            Gera Shegalov
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development