Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3901

lazy load JobHistory Task and TaskAttempt details

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.2
    • Component/s: jobhistoryserver, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Modified JobHistory records in YARN to lazily load job and task reports so as to improve UI response times.

      Description

      The job history UI and MRClientProtocol calls routed via JobHistory are very slow for large jobs. Some of this time is spent parsing the history file. A good chunk is spent pre-creating lots of objects which may never be used. Those can be create when required - bringing down the load times of job history pages and getJobReport etc calls to approximately the history file parse time.

      1. MR3901.txt
        169 kB
        Siddharth Seth
      2. MR3901_v3.txt
        169 kB
        Siddharth Seth
      3. MR3901_v2.txt
        169 kB
        Siddharth Seth

        Activity

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development