Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200

Replace BufferedBlockMgr with new buffer pool

    XMLWordPrintableJSON

Details

    Description

      We want to replace BufferedBlockMgr, a query-wide buffer pool, with a new BufferPool that is shared between all queries. The goals are:

      • Support for guaranteed reservations: i.e. if a reservation is granted, the buffer pool will fulfil it (unless the OS is unable to fulfill the buffer pool's memory requirements).
      • Simplified interaction between reservations and pins (if you have a reservation, you can pin, if you don't, you can't)
      • Support for increasing reservations (up to a planner-specified limit).
      • Support for smaller buffer sizes with similar performance (so we can reduce minimum memory requirement to execute spill-to-disk algorithm)
      • Support for larger buffers to support wide rows
      • Reduced reliance on TCMalloc, which isn't suited to management of large buffers (e.g. see IMPALA-2800)
      • Better transfer model for buffer pool pages, so we can implement transfer-of-ownership consistently for row batches (instead of mixing transfer with MarkNeedsToReturn()).

      Attachments

        Issue Links

          blocks

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. IMPALA-4834 All operators should operate within a memory constraint

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2800 impalad process blocked while releasing memory after a big query

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-4179 Remove RowBatch::MarkNeedsDeepCopy() memory management API

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2795 Report spilling time per operator

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2079 Don't fail when write to scratch dir results in error.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3706 Planner should set min/max per-node buffer pool reservations to optimize query performance

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. IMPALA-3265 Create a metrics to track spilling per operator

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5165 Allocate memory for all data from BufferPool

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-1382 Wasted space in buffered-tuple-stream in presence of many NULL tuples

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2167 Remove the old (unpartitioned) HJ and AGG nodes

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. IMPALA-5656 Update BlockMgr.BytesWritten reference in docs for new buffer pool

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. IMPALA-5655 Update docs for max_block_mgr_memory/buffer_pool_limit

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5701 Additional mem_limit settings for resource management (min mem, max before spill, max spill)

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue IMPALA-5161 TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5778 Clarify logging around and usage of read_size startup option

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5169 Parallelise read I/O of BufferPool::Pin()

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5229 Try using TCMalloc + Huge Pages for buffers

          • Major - Major loss of function.
          • Resolved

          Documentation - Documentation or Website IMPALA-5786 Document buffer pool changes

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-4981 COMPUTE STATS with MT_DOP=1 and tight memory limit produces spilling error

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. IMPALA-3700 Introduce upper limit to hash table size or HashBuckets for Streaming agg

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          1.
          		 Implement basic in-memory buffer pool Sub-task Resolved Tim Armstrong
          2.
          		 Add spilling support to new buffer pool Sub-task Resolved Tim Armstrong
          3.
          		 Integrate buffer pool reservations with memtrackers Sub-task Resolved Tim Armstrong
          4.
          		 Set up query mem tracker in QueryState Sub-task Resolved Tim Armstrong
          5.
          		 Clients can violate BufferPool invariants by calling ReservationTracker methods directly. Sub-task Resolved Tim Armstrong
          6.
          		 Buffer pool unpinned invariant does not take into account multiply-pinned bytes Sub-task Resolved Tim Armstrong
          7.
          		 MemTracker::EnableReservationReporting() is not thread-safe Sub-task Resolved Tim Armstrong
          8.
          		 Compute memory reservation in planner and claim atomically in Prepare() Sub-task Resolved Tim Armstrong
          9.
          		 Port spilling ExecNodes to new buffer pool Sub-task Resolved Tim Armstrong
          10.
          		 Implement scalable buffer recycling in buffer pool Sub-task Resolved Tim Armstrong
          11.
          		 Backend support for large rows Sub-task Resolved Tim Armstrong
          12.
          		 Backend support for large rows in BufferedTupleStream Sub-task Resolved Tim Armstrong
          13.
          		 Fix BufferPool handling of scratch read errors Sub-task Resolved Tim Armstrong
          14.
          		 Port relevant BufferedBlockMgr unit tests for BufferPool Sub-task Resolved Tim Armstrong
          15.
          		 Add reservation stress option for test coverage Sub-task Resolved Tim Armstrong
          16.
          		 Validate and fix spilling performance and memory usage of new buffer pool Sub-task Resolved Tim Armstrong
          17.
          		 Queries with a large number of small joins regress in terms of memory usage due to memory reservation Sub-task Resolved Tim Armstrong
          18.
          		 TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch Sub-task Resolved Tim Armstrong
          19.
          		 Account for difference between process memory consumption and memory used by queries Sub-task Resolved Tim Armstrong
          20.
          		 Clean up BufferPool profile counters Sub-task Resolved Tim Armstrong
          21.
          		 Consider reducing partition fanout in Hash Join Sub-task Resolved Tim Armstrong
          22.
          		 Partitioned aggregation node repartitions when spilled partition could fit in memory Sub-task Resolved Tim Armstrong
          23.
          		 Ensure that NAAJ works with spilling enabled and disabled. Sub-task Resolved Tim Armstrong
          24.
          		 Performance regresses on buffer pool dev branch for high-ndv aggregations Sub-task Resolved Tim Armstrong
          25.
          		 list::size() in BufferedTupleStreamV2::AdvanceWritePage() is expensive Sub-task Resolved Tim Armstrong
          26.
          		 Consider allowing configuration of buffer pool size Sub-task Resolved Tim Armstrong
          27.
          		 Track exchange node buffers memory as part of memory reservation Sub-task Resolved Tim Armstrong
          28.
          		 BufferedTupleStreamV2::CheckConsistency() is too slow for large streams with small pages in Debug build Sub-task Resolved Tim Armstrong
          29.
          		 Consider ways to reduce the accumulation of clean pages when executing large spilling queries Sub-task Resolved Tim Armstrong
          30.
          		 Eagerly release reservation in blocking nodes Sub-task Resolved Tim Armstrong
          31.
          		 Consider always reserving memory for grouping pre-aggregations Sub-task Resolved Tim Armstrong
          32.
          		 Ensure test coverage for spilling disabled for all spilling operators Sub-task Resolved Tim Armstrong
          33.
          		 Consider reducing RESERVATION_MIN_MEM_REMAINING Sub-task Resolved Tim Armstrong
          34.
          		 SET_DENY_RESERVATION_PROBABILITY debug action is not always effective Sub-task Resolved Tim Armstrong

          Activity

            People

              tarmstrong Tim Armstrong
              tarmstrong Tim Armstrong
              Votes:
              1 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: