Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2463

Job History files are not moving to done folder when job history location is hdfs location

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If "mapreduce.jobtracker.jobhistory.location" is configured as HDFS location then either during initialization of Job Tracker (while moving old job history files) or after completion of the job, history files are not moving to done and giving following exception.

       
      2011-04-29 15:27:27,813 ERROR org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history file to DONE folder.
      java.lang.IllegalArgumentException: Wrong FS: hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected: file:///
      	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
      	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
      	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
      	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
      	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
      	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
      	at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:619)
      
      
      1. MAPREDUCE-2463.patch
        3 kB
        Devaraj K
      2. MAPREDUCE-2463-1.patch
        4 kB
        Devaraj K
      3. MAPREDUCE-2463-2.patch
        4 kB
        Devaraj K

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Devaraj K
              Reporter:
              Devaraj K
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development