Details

      Description

      This is a subtask of IMPALA-3200. The first milestone we want to hit is a usable buffer pool that does not support spilling. This would include the actual buffer pool, minus the logic for flushing unpinned blocks to disk (if it runs out of buffers, we can just terminate queries).

      Initially we should just allocate memory from TCMalloc.

      It would also include the initial work to port over exec nodes to the new buffer pool: enough to run some queries using them.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-3201: buffer pool header only

        Implementation will follow in a subsequent commit.

        The BufferPool interface and reservation bookkeeping are documented.

        Includes documentation for some guarantees of new reservation mechanism
        where reservations are always guaranteed and pages can't be pinned and
        buffers can't be allocated without without a reservation. Reservations
        are tracked via a hierarchy of ReservationTrackers.

        Change-Id: Id771dea2eb4c1aa13c30d59e8b184a7d1bca8d34
        Reviewed-on: http://gerrit.cloudera.org:8080/3992
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-3201 : buffer pool header only Implementation will follow in a subsequent commit. The BufferPool interface and reservation bookkeeping are documented. Includes documentation for some guarantees of new reservation mechanism where reservations are always guaranteed and pages can't be pinned and buffers can't be allocated without without a reservation. Reservations are tracked via a hierarchy of ReservationTrackers. Change-Id: Id771dea2eb4c1aa13c30d59e8b184a7d1bca8d34 Reviewed-on: http://gerrit.cloudera.org:8080/3992 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins
        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-3201: reservation implementation for new buffer pool

        This patch implements the reservation bookkeeping logic for the new
        buffer pool. The reservations are always guaranteed and any buffer/page
        allocations require a reservation. Reservations are tracked via a
        hierarchy of ReservationTrackers.

        Eventually ReservationTrackers should become the main mechanism for
        memory accounting, once all runtime memory is handled by the buffer
        pool. In the meantime, query/process limits are enforced and memory
        is reported through the preexisting MemTracker hierarchy.

        Reservations are accounted as consumption against a MemTracker because
        the memory is committed and cannot be used for other purposes. The
        MemTracker then uses the counters from the ReservationTracker to log
        information about buffer-pool memory down to the operator level.

        Testing:
        Includes basic tests for buffer pool client registration and various
        reservation operations.

        Change-Id: I35cc89e863efb4cc506657bfdaaaf633a10bbab6
        Reviewed-on: http://gerrit.cloudera.org:8080/3993
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-3201 : reservation implementation for new buffer pool This patch implements the reservation bookkeeping logic for the new buffer pool. The reservations are always guaranteed and any buffer/page allocations require a reservation. Reservations are tracked via a hierarchy of ReservationTrackers. Eventually ReservationTrackers should become the main mechanism for memory accounting, once all runtime memory is handled by the buffer pool. In the meantime, query/process limits are enforced and memory is reported through the preexisting MemTracker hierarchy. Reservations are accounted as consumption against a MemTracker because the memory is committed and cannot be used for other purposes. The MemTracker then uses the counters from the ReservationTracker to log information about buffer-pool memory down to the operator level. Testing: Includes basic tests for buffer pool client registration and various reservation operations. Change-Id: I35cc89e863efb4cc506657bfdaaaf633a10bbab6 Reviewed-on: http://gerrit.cloudera.org:8080/3993 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins
        Hide
        tarmstrong Tim Armstrong added a comment -

        Change subject: IMPALA-3201: in-memory buffer pool implementation
        ......................................................................

        IMPALA-3201: in-memory buffer pool implementation

        This patch implements basic in-memory buffer management, with
        reservations managed by ReservationTrackers.

        Locks are fine-grained so that the buffer pool can scale to many
        concurrent queries.

        Includes basic tests for buffer pool setup, allocation and reservations.

        Change-Id: I4bda61c31cc02d26bc83c3d458c835b0984b86a0
        Reviewed-on: http://gerrit.cloudera.org:8080/4070
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - Change subject: IMPALA-3201 : in-memory buffer pool implementation ...................................................................... IMPALA-3201 : in-memory buffer pool implementation This patch implements basic in-memory buffer management, with reservations managed by ReservationTrackers. Locks are fine-grained so that the buffer pool can scale to many concurrent queries. Includes basic tests for buffer pool setup, allocation and reservations. Change-Id: I4bda61c31cc02d26bc83c3d458c835b0984b86a0 Reviewed-on: http://gerrit.cloudera.org:8080/4070 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development