Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2658

Problem running full map & reduce jobs on mrv2

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      Following the installation instructions at: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
      the randomwriter example runs successfully. However, other full map & reduce jobs (e.g. wordcount) fail with the error:

      java.lang.UnsupportedOperationException: Incompatible with LocalRunner
      at org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200)
      at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:223)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)

      The ReduceTask evaluates the isLocal flag based on the property "mapreduce.jobtracker.address", the default value for this property in mapred-default.xml is 'local' and this is the cause of the problem.

      Setting "mapreduce.jobtracker.address" in the mapred-site.xml to something other than "local" seems to solve the problem.

        Activity

        Hide
        Arun C Murthy added a comment -

        Ahmed - do you plan to update the INSTALL docs?

        Show
        Arun C Murthy added a comment - Ahmed - do you plan to update the INSTALL docs?
        Hide
        Ahmed Radwan added a comment -

        The attached patch resolves the described issue by removing "local" as the default value for the "mapreduce.jobtracker.address" property from mapred-default.xml, and also in evaluating the isLocal flag in the ReduceTask class.

        All unit tests ran successfully

        Show
        Ahmed Radwan added a comment - The attached patch resolves the described issue by removing "local" as the default value for the "mapreduce.jobtracker.address" property from mapred-default.xml, and also in evaluating the isLocal flag in the ReduceTask class. All unit tests ran successfully
        Hide
        Ahmed Radwan added a comment -

        Arun- No I thought it'll be weird to ask the user as part of the install docs to put a value for "mapreduce.jobtracker.address" where there is no jobtracker at all. The patch I submitted addresses that by modifying the mapred-default.xml and ReduceTask.java.

        Show
        Ahmed Radwan added a comment - Arun- No I thought it'll be weird to ask the user as part of the install docs to put a value for "mapreduce.jobtracker.address" where there is no jobtracker at all. The patch I submitted addresses that by modifying the mapred-default.xml and ReduceTask.java.
        Hide
        Arun C Murthy added a comment -

        Ahmad - the patch to ReduceTask isn't appropriate.

        We should consider using the serviceloader stuff rather than mapreduce.jobtracker.address in the reduce task, want to take look? Thanks.

        Show
        Arun C Murthy added a comment - Ahmad - the patch to ReduceTask isn't appropriate. We should consider using the serviceloader stuff rather than mapreduce.jobtracker.address in the reduce task, want to take look? Thanks.
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, I'll take a look. I think this will require considering the MAPREDUCE-2400 recent changes. Any other issues I should also consider?

        Show
        Ahmed Radwan added a comment - Thanks Arun, I'll take a look. I think this will require considering the MAPREDUCE-2400 recent changes. Any other issues I should also consider?
        Hide
        Arun C Murthy added a comment -

        Sorry, I missed this. It was fixed via MAPREDUCE-3004.

        Show
        Arun C Murthy added a comment - Sorry, I missed this. It was fixed via MAPREDUCE-3004 .
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, I saw it fixed but missed closing this ticket.

        Show
        Ahmed Radwan added a comment - Thanks Arun, I saw it fixed but missed closing this ticket.

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development