Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-711

Fix memory leak when not reading from inputs due to caching

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      When you are reading from inputs and caching objects with vertex scope, you don't have to read the input again when container is reused. But it allocates memory and that leaks causing OOM. KeyValueReader does not have a API to close the reader to clear allotted memory without reading from it. Also if there was a option to pre-close inputs in Processor and not fetch input at all over the wire and do shuffle/sort it would be a good optimization.

      Attachments

        1. TEZ-711.wip.4.txt
          55 kB
          Siddharth Seth
        2. TEZ-711.wip.3.txt
          28 kB
          Siddharth Seth
        3. TEZ-711.wip.2.txt
          29 kB
          Siddharth Seth
        4. TEZ-711.wip.1.txt
          10 kB
          Siddharth Seth
        5. TEZ-711.5.txt
          65 kB
          Siddharth Seth
        6. OOM-threaddump-till-TEZ-752.txt
          22 kB
          Rohini Palaniswamy
        7. OOM-threaddump-711-5-patch.txt
          22 kB
          Rohini Palaniswamy

        Issue Links

          is required by

          Sub-task - The sub-task of the issue PIG-3658 Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TEZ-388 Pig on Tez

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. TEZ-668 Allow a Processor to trigger an Input as to when to fetch/prep data.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. TEZ-919 Fix shutdown handling for Shuffle (regular and broadcast)

          • Major - Major loss of function.
          • Closed

          Activity

            People

              sseth Siddharth Seth
              rohini Rohini Palaniswamy
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: