Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3973 [Umbrella JIRA] JobHistoryServer performance improvements in YARN+MR
  3. MAPREDUCE-3966

Add separate cache for CompletedJobs with tasks loaded and without tasks loaded in JobHIstoryServer

    Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.1
    • Fix Version/s: None
    • Component/s: jobhistoryserver, mrv2
    • Labels:
      None

      Description

      After MAPREDUCE-3901, we should have separate caches for list of CompletedJob with tasks loaded and list of CompletedJob without tasks loaded. Once we have separate cache, we should increase the cache size for the list which doesn't load tasks.

        Activity

        Hide
        Siddharth Seth added a comment -

        Instead of separating the caches, we could just cache the parsed jobInfo instances (currently caching CompletedJob), and construct CompletedJob,Task etc when required. That's not too expensive and would simplify the cache.

        Show
        Siddharth Seth added a comment - Instead of separating the caches, we could just cache the parsed jobInfo instances (currently caching CompletedJob), and construct CompletedJob,Task etc when required. That's not too expensive and would simplify the cache.
        Hide
        Craig Welch added a comment -

        Hmm, I actually thing we would be better off pulling out all of this complex storage / caching logic and defer that to a general purpose database layer, as suggested here: https://issues.apache.org/jira/browse/MAPREDUCE-3973?focusedCommentId=13226450&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13226450

        Show
        Craig Welch added a comment - Hmm, I actually thing we would be better off pulling out all of this complex storage / caching logic and defer that to a general purpose database layer, as suggested here: https://issues.apache.org/jira/browse/MAPREDUCE-3973?focusedCommentId=13226450&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13226450

          People

          • Assignee:
            Unassigned
            Reporter:
            Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development