Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8789

Add BoundedQueue to AsyncDispatcher

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: applications
    • Labels:
      None
    • Flags:
      Patch

      Description

      I recently came across a scenario where an MR ApplicationMaster was failing with an OOM exception. It had many thousands of Mappers and thousands of Reducers. It was noted that in the logging that the event-queue of AsyncDispatcher had a very large number of item in it and was seemingly never decreasing.

      I started looking at the code and thought it could use some clean up, simplification, and the ability to specify a bounded queue so that any incoming events are throttled until they can be processed. This will protect the ApplicationMaster from a flood of events.

      Logging Message:
      Size of event-queue is xxx

        Attachments

        1. YARN-8789.1.patch
          34 kB
          David Mollitor
        2. YARN-8789.10.patch
          167 kB
          David Mollitor
        3. YARN-8789.12.patch
          167 kB
          David Mollitor
        4. YARN-8789.14.patch
          167 kB
          David Mollitor
        5. YARN-8789.2.patch
          36 kB
          David Mollitor
        6. YARN-8789.3.patch
          37 kB
          David Mollitor
        7. YARN-8789.4.patch
          44 kB
          David Mollitor
        8. YARN-8789.5.patch
          45 kB
          David Mollitor
        9. YARN-8789.6.patch
          45 kB
          David Mollitor
        10. YARN-8789.7.patch
          48 kB
          David Mollitor
        11. YARN-8789.7.patch
          48 kB
          David Mollitor
        12. YARN-8789.8.patch
          47 kB
          David Mollitor
        13. YARN-8789.9.patch
          53 kB
          David Mollitor

          Activity

            People

            • Assignee:
              belugabehr David Mollitor
              Reporter:
              belugabehr David Mollitor
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: