Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2350

LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.

      See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();

      Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.

      1. MAPREDUCE-2350-1.patch
        8 kB
        Devaraj K
      2. MAPREDUCE-2350.patch
        8 kB
        Devaraj K

        Issue Links

          Activity

          Ahmed Radwan created issue -
          Ahmed Radwan made changes -
          Field Original Value New Value
          Summary LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of wether the old API is used or the new API) LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
          Devaraj K made changes -
          Attachment MAPREDUCE-2350.patch [ 12489188 ]
          Hide
          Devaraj K added a comment -

          Provided patch against trunk.

          Show
          Devaraj K added a comment - Provided patch against trunk.
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Devaraj K [ devaraj.k ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Hide
          Shai Erera added a comment -

          Thanks Devaraj for looking into this. Patch looks good to me.

          Show
          Shai Erera added a comment - Thanks Devaraj for looking into this. Patch looks good to me.
          Hide
          Ahmed Radwan added a comment -

          lgtm. Thanks Devaraj!

          Show
          Ahmed Radwan added a comment - lgtm. Thanks Devaraj!
          Hide
          Arun C Murthy added a comment -

          Sorry, the patch has gone stale. Can you please rebase? Thanks.

          Show
          Arun C Murthy added a comment - Sorry, the patch has gone stale. Can you please rebase? Thanks.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Devaraj K made changes -
          Attachment MAPREDUCE-2350-1.patch [ 12493562 ]
          Hide
          Devaraj K added a comment -

          Rebased the patch against latest directory layout.

          Show
          Devaraj K added a comment - Rebased the patch against latest directory layout.
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493562/MAPREDUCE-2350-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          org.apache.hadoop.yarn.server.nodemanager.TestNMAuditLogger

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493562/MAPREDUCE-2350-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.TestNMAuditLogger +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          This failed testcase and findbugs are not related to this patch.

          Show
          Devaraj K added a comment - This failed testcase and findbugs are not related to this patch.
          Arun C Murthy made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Hide
          Devaraj K added a comment -

          It was fixed with the jira MAPREDUCE-3563.

          Show
          Devaraj K added a comment - It was fixed with the jira MAPREDUCE-3563 .
          Devaraj K made changes -
          Link This issue is duplicated by MAPREDUCE-3563 [ MAPREDUCE-3563 ]
          Devaraj K made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.24.0 [ 12317654 ]
          Resolution Duplicate [ 3 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          34d 18h 30m 1 Arun C Murthy 07/Sep/11 08:57
          Open Open Patch Available Patch Available
          155d 15h 13m 2 Devaraj K 08/Sep/11 09:37
          Patch Available Patch Available Resolved Resolved
          1253d 1h 37m 1 Devaraj K 12/Feb/15 10:15

            People

            • Assignee:
              Devaraj K
              Reporter:
              Ahmed Radwan
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development