[IMPALA-2885] Scanners store per-split objects in per-query object pool

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.7.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Product Backlog
• Assignee: Michael Ho
• Reporter: Tim Armstrong
• Created: 25/Jan/16 23:06
• Updated: 26/Nov/18 22:26
• Resolved: 26/Nov/18 22:26

Description

Various scanners store control structures in RuntimeState::object_pool_ to be cleaned up at the end of the query. Since some of these control structures are allocated for every input split, a small amount of memory is wasted on control structures that are no longer needed. If a large number of scan ranges and columns are processed in a query, this can add megabytes or 10s of megabytes to the query's memory consumption.

I added some logging and saw that for a largish scan there was 10000+ objects in the object pool.

Issue Links

• is related to: IMPALA-2940 Parquet DictDecoders accumulate throughout query