Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2658

Problem running full map & reduce jobs on mrv2

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      Following the installation instructions at: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
      the randomwriter example runs successfully. However, other full map & reduce jobs (e.g. wordcount) fail with the error:

      java.lang.UnsupportedOperationException: Incompatible with LocalRunner
      at org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200)
      at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:223)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)

      The ReduceTask evaluates the isLocal flag based on the property "mapreduce.jobtracker.address", the default value for this property in mapred-default.xml is 'local' and this is the cause of the problem.

      Setting "mapreduce.jobtracker.address" in the mapred-site.xml to something other than "local" seems to solve the problem.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        8d 1h 52m 1 Ahmed Radwan 15/Jul/11 22:44
        Patch Available Patch Available Open Open
        3m 38s 1 Arun C Murthy 15/Jul/11 22:47
        Open Open Resolved Resolved
        68d 22h 37m 1 Arun C Murthy 22/Sep/11 21:25
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, I saw it fixed but missed closing this ticket.

        Show
        Ahmed Radwan added a comment - Thanks Arun, I saw it fixed but missed closing this ticket.
        Arun C Murthy made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Duplicate [ 3 ]
        Hide
        Arun C Murthy added a comment -

        Sorry, I missed this. It was fixed via MAPREDUCE-3004.

        Show
        Arun C Murthy added a comment - Sorry, I missed this. It was fixed via MAPREDUCE-3004 .
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, I'll take a look. I think this will require considering the MAPREDUCE-2400 recent changes. Any other issues I should also consider?

        Show
        Ahmed Radwan added a comment - Thanks Arun, I'll take a look. I think this will require considering the MAPREDUCE-2400 recent changes. Any other issues I should also consider?
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Arun C Murthy added a comment -

        Ahmad - the patch to ReduceTask isn't appropriate.

        We should consider using the serviceloader stuff rather than mapreduce.jobtracker.address in the reduce task, want to take look? Thanks.

        Show
        Arun C Murthy added a comment - Ahmad - the patch to ReduceTask isn't appropriate. We should consider using the serviceloader stuff rather than mapreduce.jobtracker.address in the reduce task, want to take look? Thanks.
        Ahmed Radwan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ahmed Radwan added a comment -

        Arun- No I thought it'll be weird to ask the user as part of the install docs to put a value for "mapreduce.jobtracker.address" where there is no jobtracker at all. The patch I submitted addresses that by modifying the mapred-default.xml and ReduceTask.java.

        Show
        Ahmed Radwan added a comment - Arun- No I thought it'll be weird to ask the user as part of the install docs to put a value for "mapreduce.jobtracker.address" where there is no jobtracker at all. The patch I submitted addresses that by modifying the mapred-default.xml and ReduceTask.java.
        Ahmed Radwan made changes -
        Attachment MAPREDUCE-2658.patch [ 12486695 ]
        Hide
        Ahmed Radwan added a comment -

        The attached patch resolves the described issue by removing "local" as the default value for the "mapreduce.jobtracker.address" property from mapred-default.xml, and also in evaluating the isLocal flag in the ReduceTask class.

        All unit tests ran successfully

        Show
        Ahmed Radwan added a comment - The attached patch resolves the described issue by removing "local" as the default value for the "mapreduce.jobtracker.address" property from mapred-default.xml, and also in evaluating the isLocal flag in the ReduceTask class. All unit tests ran successfully
        Hide
        Arun C Murthy added a comment -

        Ahmed - do you plan to update the INSTALL docs?

        Show
        Arun C Murthy added a comment - Ahmed - do you plan to update the INSTALL docs?
        Ahmed Radwan made changes -
        Field Original Value New Value
        Assignee Ahmed Radwan [ ahmed.radwan ]
        Ahmed Radwan created issue -

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development