[IMPALA-2506] Account for memory usage when preparing fragment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.3.0
• Component/s: Backend
• Labels: query-lifecycle resource-management
• Target Version: Product Backlog
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 08/Oct/15 00:07
• Updated: 02/Nov/16 15:39
• Resolved: 07/Jun/16 16:12

Description

Preparing a query fragment for execution can consume a significant amount of untracked memory, between plan fragments, descriptors, and in-memory LLVM structures. This is problematic if we're at or near the process memory limit, since it will go further over the limit. This memory isn't freed until the fragment is fully canceled and deregistered.

In this situation better behaviour would be to not start the fragment and notify the coordinator that the fragment was cancelled immediately to avoid consuming resources unnecessarily.

Issue Links

• is related to: IMPALA-3290 Catch std::bad_alloc from LLVM optimisation
• is related to: IMPALA-3091 LLVM memory is not accounted for in query mem tracker
• is related to: IMPALA-2120 Track mem of std structures in query execution