Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3901

lazy load JobHistory Task and TaskAttempt details

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.2
    • Component/s: jobhistoryserver, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Modified JobHistory records in YARN to lazily load job and task reports so as to improve UI response times.

      Description

      The job history UI and MRClientProtocol calls routed via JobHistory are very slow for large jobs. Some of this time is spent parsing the history file. A good chunk is spent pre-creating lots of objects which may never be used. Those can be create when required - bringing down the load times of job history pages and getJobReport etc calls to approximately the history file parse time.

      1. MR3901.txt
        169 kB
        Siddharth Seth
      2. MR3901_v2.txt
        169 kB
        Siddharth Seth
      3. MR3901_v3.txt
        169 kB
        Siddharth Seth

        Activity

        Siddharth Seth created issue -
        Siddharth Seth made changes -
        Field Original Value New Value
        Attachment MR3901.txt [ 12515687 ]
        Siddharth Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siddharth Seth made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siddharth Seth made changes -
        Attachment MR3901_v2.txt [ 12515794 ]
        Siddharth Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siddharth Seth made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Siddharth Seth made changes -
        Attachment MR3901_v3.txt [ 12515820 ]
        Siddharth Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Release Note Modified JobHistory records in YARN to lazily load job and task reports so as to improve UI response times.
        Fix Version/s 0.23.2 [ 12319851 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development