Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3700

Introduce upper limit to hash table size or HashBuckets for Streaming agg

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • Impala 2.7.0
    • None
    • Backend

    Description

      Several queries consume all the memory limit in Streaming agg leaving very little memory left for other operators.

      Attached is the profile for TPC-DS Q67 which fails for the reason mentioned.

      An alternative solution is to extend the operator interface to respond to memory pressure by spilling and releasing some memory back to the memory/block manager.

      Attachments

        1. query67.sql.1.out
          10 kB
          Mostafa Mokhtar
        2. query67.sql.1.profile.out
          5.33 MB
          Mostafa Mokhtar

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3706 Planner should set min/max per-node buffer pool reservations to optimize query performance

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-1346 Sorter fails with out-of-memory in queries with multiple spilling operators

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              mmokhtar Mostafa Mokhtar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: