Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-863 Improve/standardize job history file content format and the management of the history files
  3. MAPREDUCE-817

Add a cache for retired jobs with minimal job info and provide a way to access history file url

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.21.0
    • client, jobtracker
    • None
    • Incompatible change, Reviewed
    • Hide
      Provides a way to configure the cache of JobStatus objects for the retired jobs.
      Adds an API in RunningJob to access history file url.
      Adds a LRU based cache for job history files loaded in memory when accessed via JobTracker web UI.
      Adds Retired Jobs table on the Jobtracker UI. The job move from Running to Completed/Failed table. Then job move to Retired table when it is purged from memory. The Retired table shows last 100 retired jobs. The Completed/Failed jobs table are only shown if there are non-zero jobs in the table.
      Show
      Provides a way to configure the cache of JobStatus objects for the retired jobs. Adds an API in RunningJob to access history file url. Adds a LRU based cache for job history files loaded in memory when accessed via JobTracker web UI. Adds Retired Jobs table on the Jobtracker UI. The job move from Running to Completed/Failed table. Then job move to Retired table when it is purged from memory. The Retired table shows last 100 retired jobs. The Completed/Failed jobs table are only shown if there are non-zero jobs in the table.

    Description

      MAPREDUCE-814 will provide a way to keep the job history files in HDFS. There should be a way to get the url for the completed job history fie. The completed jobs can be purged from memory more aggressively from jobtracker since the clients can retrieve the information from history file. Jobtracker can just maintain the very basic info about the completed jobs.

      Attachments

        1. 817_v1.patch
          60 kB
          Sharad Agarwal
        2. 817_v2.patch
          64 kB
          Sharad Agarwal
        3. 817_v3.patch
          61 kB
          Sharad Agarwal
        4. 817_ydist_new_1.patch
          2 kB
          Sharad Agarwal
        5. 817_ydist_new.patch
          38 kB
          Sharad Agarwal
        6. 817_ydist.patch
          49 kB
          Sharad Agarwal

        Issue Links

          incorporates

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-757 JobConf will not be deleted from the logs folder if job retires from finalizeJob()

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Sub-task - The sub-task of the issue MAPREDUCE-814 Move completed Job history files to HDFS

          • Major - Major loss of function.
          • Closed

          Activity

            People

              sharadag Sharad Agarwal
              sharadag Sharad Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: