Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5220

Clean up TCMalloc memory maintenance logic

    Details

      Description

      As mentioned in IMPALA-2800, TCMalloc's default behaviour so that it does not keep unmapped memory around. This means that our various ReleaseMemory()/ReleaseToSystem() calls may not have any effect.

      We should verify this and then remove any logic that is no longer needed.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5220: memory maintenance cleanup

        Remove logic that tries to release pages from TcMalloc's page heap:
        TCMalloc's behaviour changed so that it automatically does this with
        "aggressive decommit" and committed spans can't accumulate in the page
        heap.

        Also increase the memory maintenance interval - 1s is too aggressive and
        can free memory that will be imminently reused by a running query, e.g.
        particularly buffer pool buffers.

        Testing:
        Tried to reproduce IMPALA-2800 in a couple of ways:

        • Ran a big aggregation locally and cancelled it
        • Looked at memz/ of some live clusters (production and stress test).
          In all cases "Bytes in page heap freelist" was 0.

        This confirms that IMPALA-2800 was already solved, probably
        by the gperftools 2.5 upgrade, where aggressive decommit
        would mean that memory is released to the system in
        free() instead of the ReleaseFreeMemory() callst.

        I was able to confirm that the ReleaseFreeMemory() calls were unnecessary
        to avoid retaining memory by running a couple of stress tests
        locally with this patch and checking that "Bytes in page
        heap freelist" was 0 after the change and that memory consumption
        was generally sensible.

        Change-Id: I0f822b294ab253d6f2828fc52f353aecaaf9b701
        Reviewed-on: http://gerrit.cloudera.org:8080/6626
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5220 : memory maintenance cleanup Remove logic that tries to release pages from TcMalloc's page heap: TCMalloc's behaviour changed so that it automatically does this with "aggressive decommit" and committed spans can't accumulate in the page heap. Also increase the memory maintenance interval - 1s is too aggressive and can free memory that will be imminently reused by a running query, e.g. particularly buffer pool buffers. Testing: Tried to reproduce IMPALA-2800 in a couple of ways: Ran a big aggregation locally and cancelled it Looked at memz/ of some live clusters (production and stress test). In all cases "Bytes in page heap freelist" was 0. This confirms that IMPALA-2800 was already solved, probably by the gperftools 2.5 upgrade, where aggressive decommit would mean that memory is released to the system in free() instead of the ReleaseFreeMemory() callst. I was able to confirm that the ReleaseFreeMemory() calls were unnecessary to avoid retaining memory by running a couple of stress tests locally with this patch and checking that "Bytes in page heap freelist" was 0 after the change and that memory consumption was generally sensible. Change-Id: I0f822b294ab253d6f2828fc52f353aecaaf9b701 Reviewed-on: http://gerrit.cloudera.org:8080/6626 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins —

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development