Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3954

Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.2, 0.10.0
    • None
    • None

    Description

      Tez Shuffle Handler holds on to entire SpillRecord to better accommodate for range fetch operation. For single reduce fetch holding on to spillRecord can overwhelm memory if Index Cache is overrun. In the case of single fetch, only keep the single tez index record needed and for range fetch, keep the entire spill record in memory, but free it after it is needed.

      Attachments

        1. TEZ-3954.001.patch
          7 kB
          Jonathan Turner Eagles

        Activity

          People

            jeagles Jonathan Turner Eagles
            jeagles Jonathan Turner Eagles
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: