Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2658

Problem running full map & reduce jobs on mrv2

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      Following the installation instructions at: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
      the randomwriter example runs successfully. However, other full map & reduce jobs (e.g. wordcount) fail with the error:

      java.lang.UnsupportedOperationException: Incompatible with LocalRunner
      at org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200)
      at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:223)
      at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)

      The ReduceTask evaluates the isLocal flag based on the property "mapreduce.jobtracker.address", the default value for this property in mapred-default.xml is 'local' and this is the cause of the problem.

      Setting "mapreduce.jobtracker.address" in the mapred-site.xml to something other than "local" seems to solve the problem.

        Activity

        Ahmed Radwan created issue -
        Ahmed Radwan made changes -
        Field Original Value New Value
        Assignee Ahmed Radwan [ ahmed.radwan ]
        Ahmed Radwan made changes -
        Attachment MAPREDUCE-2658.patch [ 12486695 ]
        Ahmed Radwan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Duplicate [ 3 ]

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development