Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 0.23.5
    • Fix Version/s: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
    • Component/s: mr-am
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The AM does not have any flow control to limit the incoming rate of events from tasks. If the AM is unable to keep pace with the rate of incoming events for a sufficient period of time then it will eventually exhaust the heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event processing, but the AM could still get behind if it's starved for CPU and/or handling a very large job with tens of thousands of active tasks.

        Attachments

        1. MAPREDUCE-5124-branch-2.7.002.patch
          29 kB
          Jason Lowe
        2. MAPREDUCE-5124-branch-2.8.002.patch
          30 kB
          Jason Lowe
        3. MAPREDUCE-5124-branch-2.7.002.patch
          29 kB
          Peter Bacsko
        4. MAPREDUCE-5124-branch-2.9.002.patch
          30 kB
          Peter Bacsko
        5. MAPREDUCE-5124-branch-2.8.002.patch
          30 kB
          Peter Bacsko
        6. MAPREDUCE-5124-branch-2.002.patch
          30 kB
          Peter Bacsko
        7. MAPREDUCE-5124-branch-2.8.001.patch
          30 kB
          Jason Lowe
        8. MAPREDUCE-5124-branch-2.9.001.patch
          30 kB
          Peter Bacsko
        9. MAPREDUCE-5124-branch-2.001.patch
          30 kB
          Peter Bacsko
        10. MAPREDUCE-5124-branch-2.8.001.patch
          30 kB
          Peter Bacsko
        11. MAPREDUCE-5124-branch-2.7.001.patch
          30 kB
          Peter Bacsko
        12. MAPREDUCE-5124-003.patch
          34 kB
          Peter Bacsko
        13. MAPREDUCE-5124-002.patch
          34 kB
          Peter Bacsko
        14. MAPREDUCE-5124-001.patch
          33 kB
          Peter Bacsko
        15. MAPREDUCE-5124-CoalescingPOC3.patch
          12 kB
          Peter Bacsko
        16. MAPREDUCE-5124-CoalescingPOC2.patch
          13 kB
          Peter Bacsko
        17. MAPREDUCE-5124-CoalescingPOC-1.patch
          13 kB
          Peter Bacsko
        18. MAPREDUCE-5124-proto.2.txt
          15 kB
          Tsuyoshi Ozawa
        19. MAPREDUCE-5124-prototype.txt
          9 kB
          Tsuyoshi Ozawa

          Issue Links

            Activity

              People

              • Assignee:
                pbacsko Peter Bacsko
                Reporter:
                jlowe Jason Lowe
              • Votes:
                0 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: