[IMPALA-7323] Allow closing scan nodes before all returned batches are destroyed

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend
• Labels: resource-management
• Epic Link: Simplify memory transfer model
• Target Version: Product Backlog
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 19/Jul/18 17:24
• Updated: 11/Sep/18 00:52

Description

If the scan node finishes early because of a limit at the scan or higher up in the plan tree, returned batches may still be in use by nodes higher up in the operator tree that reference memory still owned by the Scanner object, or enqueued in the RowBatchQueue. We should add a method that can be called before Close() to attach all of those resources to the trailing batch. E.g. AcquireResourcesFinal().

This method needs to be separate from Close() because Close() releases all resources accounted against the node being closed (e.g. reservations but ownership of the returned resources is only done lazily when required by a blocking node).