[MAPREDUCE-2658] Problem running full map & reduce jobs on mrv2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: mrv2
• Assignee: Ahmed Radwan
• Reporter: Ahmed Radwan
• Created: 07/Jul/11 20:51
• Updated: 22/Sep/11 21:27
• Resolved: 22/Sep/11 21:25

Description

Following the installation instructions at: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
the randomwriter example runs successfully. However, other full map & reduce jobs (e.g. wordcount) fail with the error:

java.lang.UnsupportedOperationException: Incompatible with LocalRunner
at org.apache.hadoop.mapred.YarnOutputFiles.getInputFile(YarnOutputFiles.java:200)
at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:223)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)

The ReduceTask evaluates the isLocal flag based on the property "mapreduce.jobtracker.address", the default value for this property in mapred-default.xml is 'local' and this is the cause of the problem.

Setting "mapreduce.jobtracker.address" in the mapred-site.xml to something other than "local" seems to solve the problem.