Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200

Replace BufferedBlockMgr with new buffer pool

    XMLWordPrintableJSON

    Details

      Description

      We want to replace BufferedBlockMgr, a query-wide buffer pool, with a new BufferPool that is shared between all queries. The goals are:

      • Support for guaranteed reservations: i.e. if a reservation is granted, the buffer pool will fulfil it (unless the OS is unable to fulfill the buffer pool's memory requirements).
      • Simplified interaction between reservations and pins (if you have a reservation, you can pin, if you don't, you can't)
      • Support for increasing reservations (up to a planner-specified limit).
      • Support for smaller buffer sizes with similar performance (so we can reduce minimum memory requirement to execute spill-to-disk algorithm)
      • Support for larger buffers to support wide rows
      • Reduced reliance on TCMalloc, which isn't suited to management of large buffers (e.g. see IMPALA-2800)
      • Better transfer model for buffer pool pages, so we can implement transfer-of-ownership consistently for row batches (instead of mixing transfer with MarkNeedsToReturn()).

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                1 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: