[HADOOP-2228] Jobs fail because job.xml exists

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.3
• Fix Version/s: 0.15.2, 0.16.0

Environment

35 node cluster, linux

• Assignee: Johan Oskarsson
• Reporter: Johan Oskarsson
• Created: 19/Nov/07 17:44
• Updated: 08/Jul/09 16:52
• Resolved: 15/Dec/07 16:56

Description

org.apache.hadoop.ipc.RemoteException: java.io.IOException: Target /var/storage/4/mapred/local/jobTracker/job_200711081903_3976.xml already exists
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:271)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:803)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:784)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:134)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1479)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)

at org.apache.hadoop.ipc.Client.call(Client.java:470)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:165)
at $Proxy1.submitJob(Unknown Source)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.submitJob(Unknown Source)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:397)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:345)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:250)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:282)
at java.lang.Thread.run(Thread.java:619)

Perhaps related to HADOOP-1057, HADOOP-891 or to the rpc retry. It seems my job was submitted and actually finished despite the exception. Could it be that the job went in and the rpc retry decided to submit it again anyway?