[IMPALA-4834] All operators should operate within a memory constraint

• Type: Epic
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: resource-management
• Epic Name: Operator memory constraints
• Target Version: Product Backlog
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 27/Jan/17 22:51
• Updated: 02/Oct/18 00:56

Description

Once the reservation management of IMPALA-3200 is enabled, we should start modifying more operators to operate within a memory constraint, so that once the query starts the operators can run to completion.

This means that the bulk of the operator memory, particularly the memory that depends on the input size, should be allocated from the buffer pool. The operator should then be able to execute so long as it gets it's minimum reservation.

It is ok for now if expr memory and smaller amounts of memory like rowbatches and small decompression buffers that are streamed through the plan isn't included in this.

We also don't need to handle arbitrarily sized rows, only up to a configurable threshold (e.g. a query option)

Issue Links

• depends upon: IMPALA-5485 Exchg / data-stream recv-side buffers should operate under a memory reservation
• is blocked by: IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
• is duplicated by: IMPALA-4679 Impala to set query mem_limit automatically for compute stats
• is duplicated by: IMPALA-5701 Additional mem_limit settings for resource management (min mem, max before spill, max spill)
• is related to: IMPALA-6460 More flexible memory-based admission control policies
• is related to: IMPALA-7648 Add tests for all cases where OOM is expected
• relates to: IMPALA-5165 Allocate memory for all data from BufferPool