Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2228

Jobs fail because job.xml exists

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.14.3
    • 0.15.2, 0.16.0
    • None
    • None
    • 35 node cluster, linux

    Description

      org.apache.hadoop.ipc.RemoteException: java.io.IOException: Target /var/storage/4/mapred/local/jobTracker/job_200711081903_3976.xml already exists
      at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:271)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
      at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:803)
      at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:784)
      at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:134)
      at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1479)
      at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)

      at org.apache.hadoop.ipc.Client.call(Client.java:470)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:165)
      at $Proxy1.submitJob(Unknown Source)
      at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at $Proxy1.submitJob(Unknown Source)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:397)
      at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:345)
      at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:250)
      at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:282)
      at java.lang.Thread.run(Thread.java:619)

      Perhaps related to HADOOP-1057, HADOOP-891 or to the rpc retry. It seems my job was submitted and actually finished despite the exception. Could it be that the job went in and the rpc retry decided to submit it again anyway?

      Attachments

        1. HADOOP-2228-v1.patch
          1.0 kB
          Johan Oskarsson

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            johanoskarsson Johan Oskarsson
            johanoskarsson Johan Oskarsson
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment