[HADOOP-927] MapReduce is Broken for User-Defined Classes

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.11.0
• Assignee: Unassigned
• Reporter: Albert Chern
• Created: 24/Jan/07 23:22
• Updated: 08/Jul/09 16:52
• Resolved: 15/Feb/07 21:51

Description

Checked out trunk today and all of my jobs with a user-defined class as a map output value class fail with a ClassNotFoundException. Here's the stack trace:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.StringWrapper
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:344)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:406)
at org.apache.hadoop.mapred.ReduceTaskRunner.(ReduceTaskRunner.java:339)
at org.apache.hadoop.mapred.ReduceTask.createRunner(ReduceTask.java:91)
at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:983)
at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:366)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:359)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:823)
at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:510)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:852)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1494)

I haven't looked deeply into it, but I would guess that the problem is in the constructor of ReduceTaskRunner where it attempts to create a SequenceFile.Sorter for the map output value class. It's probably done in the main TaskTracker JVM process, so external libraries are not on its classpath.

Issue Links

• duplicates: HADOOP-964 ClassNotFoundException in ReduceTaskRunner