Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5220

Clean up TCMalloc memory maintenance logic

        Details

          Description

          As mentioned in IMPALA-2800, TCMalloc's default behaviour so that it does not keep unmapped memory around. This means that our various ReleaseMemory()/ReleaseToSystem() calls may not have any effect.

          We should verify this and then remove any logic that is no longer needed.

            Activity

            Hide
            Permalink
            tarmstrong Tim Armstrong added a comment -

            IMPALA-5220: memory maintenance cleanup

            Remove logic that tries to release pages from TcMalloc's page heap:
            TCMalloc's behaviour changed so that it automatically does this with
            "aggressive decommit" and committed spans can't accumulate in the page
            heap.

            Also increase the memory maintenance interval - 1s is too aggressive and
            can free memory that will be imminently reused by a running query, e.g.
            particularly buffer pool buffers.

            Testing:
            Tried to reproduce IMPALA-2800 in a couple of ways:

            • Ran a big aggregation locally and cancelled it
            • Looked at memz/ of some live clusters (production and stress test).
              In all cases "Bytes in page heap freelist" was 0.

            This confirms that IMPALA-2800 was already solved, probably
            by the gperftools 2.5 upgrade, where aggressive decommit
            would mean that memory is released to the system in
            free() instead of the ReleaseFreeMemory() callst.

            I was able to confirm that the ReleaseFreeMemory() calls were unnecessary
            to avoid retaining memory by running a couple of stress tests
            locally with this patch and checking that "Bytes in page
            heap freelist" was 0 after the change and that memory consumption
            was generally sensible.

            Change-Id: I0f822b294ab253d6f2828fc52f353aecaaf9b701
            Reviewed-on: http://gerrit.cloudera.org:8080/6626
            Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
            Tested-by: Impala Public Jenkins
            —

            Show
            tarmstrong Tim Armstrong added a comment - IMPALA-5220 : memory maintenance cleanup Remove logic that tries to release pages from TcMalloc's page heap: TCMalloc's behaviour changed so that it automatically does this with "aggressive decommit" and committed spans can't accumulate in the page heap. Also increase the memory maintenance interval - 1s is too aggressive and can free memory that will be imminently reused by a running query, e.g. particularly buffer pool buffers. Testing: Tried to reproduce IMPALA-2800 in a couple of ways: Ran a big aggregation locally and cancelled it Looked at memz/ of some live clusters (production and stress test). In all cases "Bytes in page heap freelist" was 0. This confirms that IMPALA-2800 was already solved, probably by the gperftools 2.5 upgrade, where aggressive decommit would mean that memory is released to the system in free() instead of the ReleaseFreeMemory() callst. I was able to confirm that the ReleaseFreeMemory() calls were unnecessary to avoid retaining memory by running a couple of stress tests locally with this patch and checking that "Bytes in page heap freelist" was 0 after the change and that memory consumption was generally sensible. Change-Id: I0f822b294ab253d6f2828fc52f353aecaaf9b701 Reviewed-on: http://gerrit.cloudera.org:8080/6626 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins —

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development