Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2178

Job history on HDFS

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.17.0
    • None
    • None
    • Incompatible change
    • Hide
      This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property "mapred.job.history.location" to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property "mapred.job.history.user.location" .
      The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available.
      The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs).
      The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)
      Show
      This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property "mapred.job.history.location" to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property "mapred.job.history.user.location" . The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available. The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs). The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)

    Description

      This issue addresses the following items :

      1. Check for accuracy of job tracker history logs.

      2. After completion of the job, copy the JobHistory.log(Master index file) and the job history files to the DFS.

      3. User can load the history with commands
      bin/hadoop job -history <directory>
      or
      bin/hadoop job -history <jobid>
      This will start a stand-alone jetty and load jsps

      Attachments

        1. patch-2178.2.txt
          110 kB
          Amareshwari Sriramadasu
        2. patch-2178.txt
          110 kB
          Amareshwari Sriramadasu
        3. patch-2178.txt
          108 kB
          Amareshwari Sriramadasu
        4. patch-2178.txt
          108 kB
          Amareshwari Sriramadasu
        5. patch-2178.txt
          107 kB
          Amareshwari Sriramadasu
        6. patch-2178.txt
          105 kB
          Amareshwari Sriramadasu
        7. patch-2178.txt
          104 kB
          Amareshwari Sriramadasu

        Issue Links

          Activity

            People

              amareshwari Amareshwari Sriramadasu
              amareshwari Amareshwari Sriramadasu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: