Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
Impala 4.1.2
-
None
-
None
-
ghx-label-6
Description
Support for a query hint to specify memory limits would make implementing memory limits in applications easier where multiple queries need different memory limits but are run through the same connection, e.g.: /* +mem_limit=4g */ select * from table
—
Session settings for mem_limit are critical for getting large queries to avoid OOM when using admission control pools (for multi tenancy) and the memory estimation happens to severely under-estimate (raising the minimum memory for the pool would also reduce concurrency). However, for applications that run multiple queries through a single connection (e.g. data visualization tools) this pattern is cumbersome:
- Set the memory limit
- Run the large query
- Reset the memory limit
- Resume other queries with good estimates
Due to this, some data visualization tools only support session settings at the connection level. If you need GBs for your worst query and MBs for your average query, this greatly limits the concurrency in that pool using a 'one size fits all' memory limit or it requires maintaining many connections of varying size memory limits to attempt to use memory efficiently.
In one production use case this would limit concurrency by 10x or more.