[IMPALA-12641] Impala underestimates LLVM memory use

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.3.0
• Component/s: Backend
• Labels: codegen
• Epic Color: ghx-label-10
• Assignee: Michael Smith
• Reporter: Michael Smith
• Created: 15/Dec/23 18:45
• Updated: 15/Dec/23 19:05

Description

As I attempted to address in https://gerrit.cloudera.org/c/20697/ - and reverted in https://gerrit.cloudera.org/c/20796 - Impala currently under-estimates memory used by LLVM's Memory Manager. It allocates memory aligned to pages (usually 4KB) but currently only reports the actual memory that was requested from those pages. Trying to account for that starts to introduce significant unplanned memory and causes memory reservation failures in tests (see IMPALA-12635).

We should fix Impala's tracking of LLVM memory use along the lines of the patch 20697. It's not clear to me that the planner accounts for memory use by code generation; it looks like it needs to be updated to better plan for it.

Issue Links

• relates to: IMPALA-12635 Tests fail with "Memory is likely oversubscribed"