Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Provides an ability to move completed job history files to a HDFS location via configuring "mapred.job.tracker.history.completed.location". If the directory location does not already exist, it would be created by jobtracker.

      Description

      Currently completed job history files remain on the jobtracker node. Having the files available on HDFS will enable clients to access these files more easily.

      1. 814_ydist.patch
        28 kB
        Sharad Agarwal
      2. 814_v5.patch
        26 kB
        Sharad Agarwal
      3. 814_v4.patch
        27 kB
        Sharad Agarwal
      4. 814_v3.patch
        23 kB
        Sharad Agarwal
      5. 814_v2.patch
        11 kB
        Sharad Agarwal
      6. 814_v1.patch
        6 kB
        Sharad Agarwal

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/ )
          Hide
          Sharad Agarwal added a comment -

          Patch for Yahoo's distribution.

          Show
          Sharad Agarwal added a comment - Patch for Yahoo's distribution.
          Hide
          Sharad Agarwal added a comment -

          I just committed this.

          Show
          Sharad Agarwal added a comment - I just committed this.
          Hide
          Devaraj Das added a comment -

          +1

          Show
          Devaraj Das added a comment - +1
          Hide
          Sharad Agarwal added a comment -

          test patch and ant test passed.

          Show
          Sharad Agarwal added a comment - test patch and ant test passed.
          Hide
          Sharad Agarwal added a comment -

          Incorporated Devaraj's offline comments. Minimized the jobtracker init changes. Passing filesystem handle in JobHistory#getJobHistoryFileName

          Show
          Sharad Agarwal added a comment - Incorporated Devaraj's offline comments. Minimized the jobtracker init changes. Passing filesystem handle in JobHistory#getJobHistoryFileName
          Hide
          Sharad Agarwal added a comment -

          ant test and test patch passed.

          +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Sharad Agarwal added a comment - ant test and test patch passed. +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Sharad Agarwal added a comment -

          Changes from last patch: Refactored JobTracker constructor to fix an issue related to Jobtracker recovery requiring history to be inited.

          Show
          Sharad Agarwal added a comment - Changes from last patch: Refactored JobTracker constructor to fix an issue related to Jobtracker recovery requiring history to be inited.
          Hide
          Sharad Agarwal added a comment -

          Retrying Hudswon

          Show
          Sharad Agarwal added a comment - Retrying Hudswon
          Hide
          Sharad Agarwal added a comment -

          Fixed issue related to filesystem handle in JobHistory. Now log dir and done dir could have different filesystem.
          Added a testcase to have done folder on HDFS.

          Show
          Sharad Agarwal added a comment - Fixed issue related to filesystem handle in JobHistory. Now log dir and done dir could have different filesystem. Added a testcase to have done folder on HDFS.
          Hide
          Sharad Agarwal added a comment -

          Added a testcase and tested manually locally.

          Show
          Sharad Agarwal added a comment - Added a testcase and tested manually locally.
          Hide
          Sharad Agarwal added a comment -

          An early patch:
          -Moves the history files to HDFS in a separate thread. I have used a thread pool so that the movement of files are not stalled for long time in case of datanodes being slow.
          -Users can provide the location of the done folder using "mapred.job.tracker.history.completed.location"
          -Testing in progress. Test case not added yet.

          Show
          Sharad Agarwal added a comment - An early patch: -Moves the history files to HDFS in a separate thread. I have used a thread pool so that the movement of files are not stalled for long time in case of datanodes being slow. -Users can provide the location of the done folder using "mapred.job.tracker.history.completed.location" -Testing in progress. Test case not added yet.
          Hide
          Sharad Agarwal added a comment -

          @Amar I would prefer to keep the scope of this Jira for just providing a way to move files to HDFS. Job retire can perhaps be discussed in MAPREDUCE-817

          Show
          Sharad Agarwal added a comment - @Amar I would prefer to keep the scope of this Jira for just providing a way to move files to HDFS. Job retire can perhaps be discussed in MAPREDUCE-817
          Hide
          Amar Kamat added a comment -

          This is the right place to refactor job retire.

          Show
          Amar Kamat added a comment - This is the right place to refactor job retire.

            People

            • Assignee:
              Sharad Agarwal
              Reporter:
              Sharad Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development