[IMPALA-3200] Replace BufferedBlockMgr with new buffer pool

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.10.0
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 18/Mar/16 00:06
• Updated: 10/Aug/18 16:36
• Resolved: 25/Aug/17 00:54

Description

We want to replace BufferedBlockMgr, a query-wide buffer pool, with a new BufferPool that is shared between all queries. The goals are:

• Support for guaranteed reservations: i.e. if a reservation is granted, the buffer pool will fulfil it (unless the OS is unable to fulfill the buffer pool's memory requirements).
• Simplified interaction between reservations and pins (if you have a reservation, you can pin, if you don't, you can't)
• Support for increasing reservations (up to a planner-specified limit).
• Support for smaller buffer sizes with similar performance (so we can reduce minimum memory requirement to execute spill-to-disk algorithm)
• Support for larger buffers to support wide rows
• Reduced reliance on TCMalloc, which isn't suited to management of large buffers (e.g. see IMPALA-2800)
• Better transfer model for buffer pool pages, so we can implement transfer-of-ownership consistently for row batches (instead of mixing transfer with MarkNeedsToReturn()).

Issue Links

• blocks: IMPALA-4834 All operators should operate within a memory constraint
• blocks: IMPALA-2800 impalad process blocked while releasing memory after a big query
• blocks: IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space
• blocks: IMPALA-4179 Remove RowBatch::MarkNeedsDeepCopy() memory management API
• blocks: IMPALA-2795 Report spilling time per operator
• blocks: IMPALA-2079 Don't fail when write to scratch dir results in error.
• blocks: IMPALA-3706 Planner should set min/max per-node buffer pool reservations to optimize query performance
• blocks: IMPALA-3265 Create a metrics to track spilling per operator
• is depended upon by: IMPALA-5165 Allocate memory for all data from BufferPool
• is depended upon by: IMPALA-1382 Wasted space in buffered-tuple-stream in presence of many NULL tuples
• is depended upon by: IMPALA-2167 Remove the old (unpartitioned) HJ and AGG nodes
• is depended upon by: IMPALA-5656 Update BlockMgr.BytesWritten reference in docs for new buffer pool
• is depended upon by: IMPALA-5655 Update docs for max_block_mgr_memory/buffer_pool_limit
• is duplicated by: IMPALA-5701 Additional mem_limit settings for resource management (min mem, max before spill, max spill)
• is related to: IMPALA-5161 TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch
• is related to: IMPALA-5778 Clarify logging around and usage of read_size startup option
• is related to: IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space
• is related to: IMPALA-5169 Parallelise read I/O of BufferPool::Pin()
• is related to: IMPALA-5229 Try using TCMalloc + Huge Pages for buffers
• is related to: IMPALA-5786 Document buffer pool changes
• is related to: IMPALA-4981 COMPUTE STATS with MT_DOP=1 and tight memory limit produces spilling error
• relates to: IMPALA-3700 Introduce upper limit to hash table size or HashBuckets for Streaming agg