[HADOOP-12] InputFormat used in job must be in JobTracker classpath (not loaded from job JAR)

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.1.0

Environment

~20 node nutch mapreduce environment, running SVN trunk, on Linux

• Assignee: Unassigned
• Reporter: Bryan Pendleton
• Created: 31/Jan/06 07:00
• Updated: 08/Jul/09 16:51
• Resolved: 10/Feb/06 08:01

Description

During development, I've been creating/tweaking custom InputFormat implementations. However, when you try to run a job against a running cluster, you get:
Exception in thread "main" java.io.IOException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: my.custom.InputFormat
at org.apache.nutch.ipc.Client.call(Client.java:294)
at org.apache.nutch.ipc.RPC$Invoker.invoke(RPC.java:127)
at $Proxy0.submitJob(Unknown Source)
at org.apache.nutch.mapred.JobClient.submitJob(JobClient.java:259)
at org.apache.nutch.mapred.JobClient.runJob(JobClient.java:288)
at com.parc.uir.wikipedia.WikipediaJob.main(WikipediaJob.java:85)

This error goes away if I restart the TaskTrackers/JobTracker with a classpath which includes the needed code. Other classes (Mapper, Reducer) appear to be available out of the jar file specified in the JobConf, but not the InputFormat. Obviously, it's less than idea to have to restart the JobTracker whenever there's a change to a job-specific class.

Issue Links

• is related to: HADOOP-16 RPC call times out while indexing map task is computing splits