Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property "mapred.job.history.location" to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property "mapred.job.history.user.location" .
      The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available.
      The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs).
      The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)
      Show
      This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property "mapred.job.history.location" to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property "mapred.job.history.user.location" . The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available. The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs). The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)

      Description

      This issue addresses the following items :

      1. Check for accuracy of job tracker history logs.

      2. After completion of the job, copy the JobHistory.log(Master index file) and the job history files to the DFS.

      3. User can load the history with commands
      bin/hadoop job -history <directory>
      or
      bin/hadoop job -history <jobid>
      This will start a stand-alone jetty and load jsps

        Attachments

        1. patch-2178.txt
          110 kB
          Amareshwari Sriramadasu
        2. patch-2178.2.txt
          110 kB
          Amareshwari Sriramadasu
        3. patch-2178.txt
          108 kB
          Amareshwari Sriramadasu
        4. patch-2178.txt
          108 kB
          Amareshwari Sriramadasu
        5. patch-2178.txt
          107 kB
          Amareshwari Sriramadasu
        6. patch-2178.txt
          105 kB
          Amareshwari Sriramadasu
        7. patch-2178.txt
          104 kB
          Amareshwari Sriramadasu

          Issue Links

            Activity

              People

              • Assignee:
                amareshwari Amareshwari Sriramadasu
                Reporter:
                amareshwari Amareshwari Sriramadasu
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: