Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2658

Problem running full map & reduce jobs on mrv2

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      Following the installation instructions at: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
      the randomwriter example runs successfully. However, other full map & reduce jobs (e.g. wordcount) fail with the error:

      java.lang.UnsupportedOperationException: Incompatible with LocalRunner
      at org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200)
      at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:223)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)

      The ReduceTask evaluates the isLocal flag based on the property "mapreduce.jobtracker.address", the default value for this property in mapred-default.xml is 'local' and this is the cause of the problem.

      Setting "mapreduce.jobtracker.address" in the mapred-site.xml to something other than "local" seems to solve the problem.

        Activity

        Ahmed Radwan created issue -
        Ahmed Radwan made changes -
        Field Original Value New Value
        Assignee Ahmed Radwan [ ahmed.radwan ]
        Hide
        Arun C Murthy added a comment -

        Ahmed - do you plan to update the INSTALL docs?

        Show
        Arun C Murthy added a comment - Ahmed - do you plan to update the INSTALL docs?
        Hide
        Ahmed Radwan added a comment -

        The attached patch resolves the described issue by removing "local" as the default value for the "mapreduce.jobtracker.address" property from mapred-default.xml, and also in evaluating the isLocal flag in the ReduceTask class.

        All unit tests ran successfully

        Show
        Ahmed Radwan added a comment - The attached patch resolves the described issue by removing "local" as the default value for the "mapreduce.jobtracker.address" property from mapred-default.xml, and also in evaluating the isLocal flag in the ReduceTask class. All unit tests ran successfully
        Ahmed Radwan made changes -
        Attachment MAPREDUCE-2658.patch [ 12486695 ]
        Hide
        Ahmed Radwan added a comment -

        Arun- No I thought it'll be weird to ask the user as part of the install docs to put a value for "mapreduce.jobtracker.address" where there is no jobtracker at all. The patch I submitted addresses that by modifying the mapred-default.xml and ReduceTask.java.

        Show
        Ahmed Radwan added a comment - Arun- No I thought it'll be weird to ask the user as part of the install docs to put a value for "mapreduce.jobtracker.address" where there is no jobtracker at all. The patch I submitted addresses that by modifying the mapred-default.xml and ReduceTask.java.
        Ahmed Radwan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Arun C Murthy added a comment -

        Ahmad - the patch to ReduceTask isn't appropriate.

        We should consider using the serviceloader stuff rather than mapreduce.jobtracker.address in the reduce task, want to take look? Thanks.

        Show
        Arun C Murthy added a comment - Ahmad - the patch to ReduceTask isn't appropriate. We should consider using the serviceloader stuff rather than mapreduce.jobtracker.address in the reduce task, want to take look? Thanks.
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, I'll take a look. I think this will require considering the MAPREDUCE-2400 recent changes. Any other issues I should also consider?

        Show
        Ahmed Radwan added a comment - Thanks Arun, I'll take a look. I think this will require considering the MAPREDUCE-2400 recent changes. Any other issues I should also consider?
        Hide
        Arun C Murthy added a comment -

        Sorry, I missed this. It was fixed via MAPREDUCE-3004.

        Show
        Arun C Murthy added a comment - Sorry, I missed this. It was fixed via MAPREDUCE-3004 .
        Arun C Murthy made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Duplicate [ 3 ]
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, I saw it fixed but missed closing this ticket.

        Show
        Ahmed Radwan added a comment - Thanks Arun, I saw it fixed but missed closing this ticket.
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        8d 1h 52m 1 Ahmed Radwan 15/Jul/11 22:44
        Patch Available Patch Available Open Open
        3m 38s 1 Arun C Murthy 15/Jul/11 22:47
        Open Open Resolved Resolved
        68d 22h 37m 1 Arun C Murthy 22/Sep/11 21:25

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development