[IMPALA-3748] Compute memory reservation in planner and claim atomically in Prepare()

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Product Backlog
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 15/Jun/16 15:36
• Updated: 12/Jul/17 04:22
• Resolved: 12/Jul/17 04:22

Description

Once we have the reservation infrastructure available, we want to make use of it by claiming memory reservations upfront in the Prepare() phase of the query.

• Compute minimum buffer pool memory requirements for each operator, and compute the peak requirement (execution of some nodes does not overlap, so this depends on the plan shape).
• Claim those reservations for the query at start of Prepare(), so we can fail fast if the memory isn't available
• Distribute the reservations to the exec nodes.

For the initial work, we can assume conservatively that all fragments run on all daemons, so that we don't need scheduling information to compute the estimates. This may over-reserve memory in some cases, since some fragments are unpartitioned or have fewer scan ranges than daemons.

Issue Links

• blocks: IMPALA-4833 Use scheduling information to make per-node memory reservation tight
• breaks: IMPALA-5231 S3 build fails because memory estimates changes
• depends upon: IMPALA-5238 Support transferring reservation between ReservationTrackers
• is blocked by: IMPALA-4862 Planner's peak resource estimates do not accurately reflect the behaviour of joins and unions in the backend