Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1986

History files fail to move to DONE folder when hadoop.job.history.location is configured to a HDFS path

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.20.2
    • None
    • jobtracker
    • None

    Description

      When hadoop.job.history.location is configured to a HDFS path and when the job retires, the history files are never moved to
      mapred.job.tracker.history.completed.location and they remain forever in hadoop.job.history.location . The jobhistory is never viewable thereafter.

      Attachments

        Issue Links

          Activity

            rramya Ramya Sunil added a comment -

            I have the following setup on my cluster:

            <property>
            <name>hadoop.job.history.location</name>
            <value>/mapred/history1</value>
            <description> If job tracker is static the history files are stored in this single well known place. If No
            value is set here, by default,
            it is in the local file system at ${hadoop.log.dir}/history.
            </description></property>

            <property>
            <name>mapred.job.tracker.history.completed.location</name>
            <value>/mapred/history2/done</value>
            <description> The completed job history files are stored at this single well known location. If nothing is
            specified, the files are stored at
            ${hadoop.job.history.location}/done.
            </description>
            </property>

            i.e. both hadoop.job.history.location and mapred.job.tracker.history.completed.location are configured to HDFS path.

            The jobtracker successfully comes up without any error. However when the job completes and it retires, I get the following exception:

            ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder.
            java.io.FileNotFoundException: File /mapred/history1/job_201007290838_0001_1280392750589_rramya_Sleep+job does not exist.
                    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
                    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)       
                    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
                    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1118)
                    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1094)
                    at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1084)
                    at org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager$1.run(JobHistory.java:300)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                    at java.lang.Thread.run(Thread.java:619)
            

            However, /mapred/history1 is created and the history files are very much present on HDFS.

            Thus the history files remain forever in hadoop.job.history.location and are never moved to mapred.job.tracker.history.completed.location. The jobhistory is never viewable thereafter.

            rramya Ramya Sunil added a comment - I have the following setup on my cluster: <property> <name>hadoop.job.history.location</name> <value>/mapred/history1</value> <description> If job tracker is static the history files are stored in this single well known place. If No value is set here, by default, it is in the local file system at ${hadoop.log.dir}/history. </description></property> <property> <name>mapred.job.tracker.history.completed.location</name> <value>/mapred/history2/done</value> <description> The completed job history files are stored at this single well known location. If nothing is specified, the files are stored at ${hadoop.job.history.location}/done. </description> </property> i.e. both hadoop.job.history.location and mapred.job.tracker.history.completed.location are configured to HDFS path. The jobtracker successfully comes up without any error. However when the job completes and it retires, I get the following exception: ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder. java.io.FileNotFoundException: File /mapred/history1/job_201007290838_0001_1280392750589_rramya_Sleep+job does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1118) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1094) at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1084) at org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager$1.run(JobHistory.java:300) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) However, /mapred/history1 is created and the history files are very much present on HDFS. Thus the history files remain forever in hadoop.job.history.location and are never moved to mapred.job.tracker.history.completed.location. The jobhistory is never viewable thereafter.

            Ramya,
            Refer MAPREDUCE-2463. This issue also address the same.

            kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment - Ramya, Refer MAPREDUCE-2463 . This issue also address the same.
            rramya Ramya Sunil added a comment -

            This issue is exactly the same as MAPREDUCE-2463. Since there is already a patch available in MAPREDUCE-2463, resolving this as duplicate. Thanks Bhallamudi.

            rramya Ramya Sunil added a comment - This issue is exactly the same as MAPREDUCE-2463 . Since there is already a patch available in MAPREDUCE-2463 , resolving this as duplicate. Thanks Bhallamudi.

            People

              Unassigned Unassigned
              rramya Ramya Sunil
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: