Details

      Description

      The reservation mechanism for the buffer pool introduces a memory tracking mechanism that is somewhat parallel to the existing MemTracker mechanism. This task is to integrate the two so that:

      • Query memory limits include buffer reservations
      • Runtime profiles and other diagnostics include buffer pool usage in the memory consumption, down to the exec node level.

      My initial thought is that each reservation tracker should optionally have an associated MemTracker. When reservations are granted, it would be counted towards the memory consumption of the tracker. I think we want to count reservations rather than usage, because the MemTrackers are really more geared towards tracking committed memory rather than used memory.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        This was solved in

        IMPALA-3201: reservation implementation for new buffer pool

        This patch implements the reservation bookkeeping logic for the new
        buffer pool. The reservations are always guaranteed and any buffer/page
        allocations require a reservation. Reservations are tracked via a
        hierarchy of ReservationTrackers.

        Eventually ReservationTrackers should become the main mechanism for
        memory accounting, once all runtime memory is handled by the buffer
        pool. In the meantime, query/process limits are enforced and memory
        is reported through the preexisting MemTracker hierarchy.

        Reservations are accounted as consumption against a MemTracker because
        the memory is committed and cannot be used for other purposes. The
        MemTracker then uses the counters from the ReservationTracker to log
        information about buffer-pool memory down to the operator level.

        Testing:
        Includes basic tests for buffer pool client registration and various
        reservation operations.

        Change-Id: I35cc89e863efb4cc506657bfdaaaf633a10bbab6
        Reviewed-on: http://gerrit.cloudera.org:8080/3993
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - This was solved in IMPALA-3201 : reservation implementation for new buffer pool This patch implements the reservation bookkeeping logic for the new buffer pool. The reservations are always guaranteed and any buffer/page allocations require a reservation. Reservations are tracked via a hierarchy of ReservationTrackers. Eventually ReservationTrackers should become the main mechanism for memory accounting, once all runtime memory is handled by the buffer pool. In the meantime, query/process limits are enforced and memory is reported through the preexisting MemTracker hierarchy. Reservations are accounted as consumption against a MemTracker because the memory is committed and cannot be used for other purposes. The MemTracker then uses the counters from the ReservationTracker to log information about buffer-pool memory down to the operator level. Testing: Includes basic tests for buffer pool client registration and various reservation operations. Change-Id: I35cc89e863efb4cc506657bfdaaaf633a10bbab6 Reviewed-on: http://gerrit.cloudera.org:8080/3993 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development