[MAPREDUCE-2463] Job History files are not moving to done folder when job history location is hdfs location

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.0
• Component/s: jobtracker
• Hadoop Flags: Reviewed
• Assignee: Devaraj Kavali
• Reporter: Devaraj Kavali
• Created: 02/May/11 06:35
• Updated: 15/Nov/11 00:49
• Resolved: 28/Jul/11 04:14

Description

If "mapreduce.jobtracker.jobhistory.location" is configured as HDFS location then either during initialization of Job Tracker (while moving old job history files) or after completion of the job, history files are not moving to done and giving following exception.

 
2011-04-29 15:27:27,813 ERROR org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history file to DONE folder.
java.lang.IllegalArgumentException: Wrong FS: hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected: file:///
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
	at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
	at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
	at org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
	at org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

Issue Links

• is duplicated by: MAPREDUCE-1986 History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path