Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5848

Account for TCMalloc overhead and client cache buffers in MemTracker

    Details

      Description

      On our stress test cluster I see a significant amount of Untracked memory. I think we can account for a chunk of it, which will further help understanding the state of a cluster.

      I see:

        Untracked Memory: Total=4.30 GB
      

      I think some of this memory is TCMalloc caches and metadata (I'm not sure exactly which is counted)

      MALLOC:    61821970968 (58958.0 MiB) Bytes in use by application
      MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
      MALLOC: +    808683664 (  771.2 MiB) Bytes in central cache freelist
      MALLOC: +      3794688 (    3.6 MiB) Bytes in transfer cache freelist
      MALLOC: +    520586840 (  496.5 MiB) Bytes in thread cache freelists
      MALLOC: +    182366400 (  173.9 MiB) Bytes in malloc metadata
      MALLOC:   ------------
      MALLOC: =  63337402560 (60403.3 MiB) Actual memory used (physical + swap)
      MALLOC: +  45341532160 (43241.1 MiB) Bytes released to OS (aka unmapped)
      MALLOC:   ------------
      MALLOC: = 108678934720 (103644.3 MiB) Virtual address space used
      MALLOC:
      MALLOC:         368033              Spans in use
      MALLOC:           3511              Thread heaps in use
      MALLOC:           8192              Tcmalloc page size
      ------------------------------------------------
      Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
      Bytes released to the OS take up virtual address space but no physical memory.
      

      The client cache also is likely taking up a significant amount of memory, since I believe the clients all have buffers attached:

      impala-server.backends.client-cache.total-clients 	1006 
      

        Attachments

          Activity

            People

            • Assignee:
              bikramjeet.vig Bikramjeet Vig
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: