Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3831

Reduce Unordered memory needed for storing empty completed events

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.1
    • Component/s: None
    • Labels:
      None

      Description

      the completedInputs blocking queue is used to store inputs for the UnorderedKVReader to consume. With Auto-reduce parallelism enabled and nearly all empty inputs, the reader can't prune the empty events from the blocking queue fast enough to keep up. In my scenario, an OOM occurred.

        Attachments

        1. TEZ-3831.001-addendum.patch
          1 kB
          Jonathan Turner Eagles
        2. TEZ-3831.001.patch
          2 kB
          Jonathan Turner Eagles
        3. Screen Shot 2017-09-13 at 4.55.11 PM.png
          186 kB
          Jonathan Turner Eagles

          Activity

            People

            • Assignee:
              jeagles Jonathan Turner Eagles
              Reporter:
              jeagles Jonathan Turner Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: