Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12161

Add query hints for memory limits

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • Impala 4.1.2
    • None
    • Frontend
    • None
    • ghx-label-6

    Description

      Support for a query hint to specify memory limits would make implementing memory limits in applications easier where multiple queries need different memory limits but are run through the same connection, e.g.:  /* +mem_limit=4g */ select * from table

      Session settings for mem_limit are critical for getting large queries to avoid OOM when using admission control pools (for multi tenancy) and the memory estimation happens to severely under-estimate (raising the minimum memory for the pool would also reduce concurrency).  However, for applications that run multiple queries through a single connection (e.g. data visualization tools) this pattern is cumbersome:

      1. Set the memory limit
      2. Run the large query
      3. Reset the memory limit
      4. Resume other queries with good estimates

      Due to this, some data visualization tools only support session settings at the connection level.  If you need GBs for your worst query and MBs for your average query, this greatly limits the concurrency in that pool using a 'one size fits all' memory limit or it requires maintaining many connections of varying size memory limits to attempt to use memory efficiently.

      In one production use case this would limit concurrency by 10x or more.

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            ebertp Peter Ebert
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: