Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM.

      Description

      The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
      History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.

        Attachments

        1. MAPREDUCE-3512-20120111.txt
          25 kB
          Vinod Kumar Vavilapalli
        2. MR3512.txt
          25 kB
          Siddharth Seth
        3. MR3512.txt
          9 kB
          Siddharth Seth

          Activity

            People

            • Assignee:
              sseth Siddharth Seth
              Reporter:
              sseth Siddharth Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: