Traffic Server
  1. Traffic Server
  2. TS-1006

memory management, cut down memory waste ?

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.1
    • Fix Version/s: 3.3.4
    • Component/s: Core
    • Labels:
      None

      Description

      when we review the memory usage in the production, there is something abnormal, ie, looks like TS take much memory than index data + common system waste, and here is some memory dump result by set "proxy.config.dump_mem_info_frequency"

      1, the one on a not so busy forwarding system:
      physics memory: 32G
      RAM cache: 22G
      DISK: 6140 GB
      average_object_size 64000

           allocated      |        in-use      | type size  |   free list name
      --------------------|--------------------|------------|----------------------------------
                671088640 |           37748736 |    2097152 | memory/ioBufAllocator[14]
               2248146944 |         2135949312 |    1048576 | memory/ioBufAllocator[13]
               1711276032 |         1705508864 |     524288 | memory/ioBufAllocator[12]
               1669332992 |         1667760128 |     262144 | memory/ioBufAllocator[11]
               2214592512 |         2211840000 |     131072 | memory/ioBufAllocator[10]
               2325741568 |         2323775488 |      65536 | memory/ioBufAllocator[9]
               2091909120 |         2089123840 |      32768 | memory/ioBufAllocator[8]
               1956642816 |         1956478976 |      16384 | memory/ioBufAllocator[7]
               2094530560 |         2094071808 |       8192 | memory/ioBufAllocator[6]
                356515840 |          355540992 |       4096 | memory/ioBufAllocator[5]
                  1048576 |              14336 |       2048 | memory/ioBufAllocator[4]
                   131072 |                  0 |       1024 | memory/ioBufAllocator[3]
                    65536 |                  0 |        512 | memory/ioBufAllocator[2]
                    32768 |                  0 |        256 | memory/ioBufAllocator[1]
                    16384 |                  0 |        128 | memory/ioBufAllocator[0]
                        0 |                  0 |        576 | memory/ICPRequestCont_allocator
                        0 |                  0 |        112 | memory/ICPPeerReadContAllocator
                        0 |                  0 |        432 | memory/PeerReadDataAllocator
                        0 |                  0 |         32 | memory/MIMEFieldSDKHandle
                        0 |                  0 |        240 | memory/INKVConnAllocator
                        0 |                  0 |         96 | memory/INKContAllocator
                     4096 |                  0 |         32 | memory/apiHookAllocator
                        0 |                  0 |        288 | memory/FetchSMAllocator
                        0 |                  0 |         80 | memory/prefetchLockHandlerAllocator
                        0 |                  0 |        176 | memory/PrefetchBlasterAllocator
                        0 |                  0 |         80 | memory/prefetchUrlBlaster
                        0 |                  0 |         96 | memory/blasterUrlList
                        0 |                  0 |         96 | memory/prefetchUrlEntryAllocator
                        0 |                  0 |        128 | memory/socksProxyAllocator
                        0 |                  0 |        144 | memory/ObjectReloadCont
                  3258368 |             576016 |        592 | memory/httpClientSessionAllocator
                   825344 |             139568 |        208 | memory/httpServerSessionAllocator
                 22597632 |            1284848 |       9808 | memory/httpSMAllocator
                        0 |                  0 |         32 | memory/CacheLookupHttpConfigAllocator
                        0 |                  0 |       9856 | memory/httpUpdateSMAllocator
                        0 |                  0 |        128 | memory/RemapPluginsAlloc
                        0 |                  0 |         48 | memory/CongestRequestParamAllocator
                        0 |                  0 |        128 | memory/CongestionDBContAllocator
                  5767168 |             704512 |       2048 | memory/hdrStrHeap
                 18350080 |            1153024 |       2048 | memory/hdrHeap
                    53248 |               2912 |        208 | memory/httpCacheAltAllocator
                        0 |                  0 |        112 | memory/OneWayTunnelAllocator
                   157696 |               1232 |       1232 | memory/hostDBContAllocator
                   102240 |              17040 |      17040 | memory/dnsBufAllocator
                   323584 |                  0 |       1264 | memory/dnsEntryAllocator
                        0 |                  0 |         16 | memory/DNSRequestDataAllocator
                        0 |                  0 |       1072 | memory/SRVAllocator
                        0 |                  0 |         48 | memory/ClusterVConnectionCache::Entry
                        0 |                  0 |        560 | memory/cacheContAllocator
                        0 |                  0 |        112 | memory/inControlAllocator
                        0 |                  0 |        112 | memory/outControlAllocator
                        0 |                  0 |         32 | memory/byteBankAllocator
                        0 |                  0 |        576 | memory/clusterVCAllocator
                        0 |                  0 |         48 | memory/evacuationKey
                     6144 |                  0 |         48 | memory/cacheRemoveCont
                   270336 |             262560 |         96 | memory/evacuationBlock
                  4997120 |            3968416 |        976 | memory/cacheVConnection
                   798720 |             522080 |        160 | memory/openDirEntry
                        0 |                  0 |         64 | memory/RamCacheLRUEntry
                 56426496 |           56426304 |         96 | memory/RamCacheCLFUSEntry
                  9584640 |            6168000 |        960 | memory/netVCAllocator
                        0 |                  0 |        128 | memory/udpReadContAllocator
                        0 |                  0 |        128 | memory/udpWorkContinuationAllocator
                        0 |                  0 |        160 | memory/udpPacketAllocator
                        0 |                  0 |        304 | memory/socksAllocator
                   139264 |              68544 |       1088 | memory/sslNetVCAllocator
                        0 |                  0 |        128 | memory/UDPIOEventAllocator
                   671744 |             115520 |         64 | memory/ioBlockAllocator
                 28305408 |           28301520 |         48 | memory/ioDataAllocator
                  2273280 |             406320 |        240 | memory/ioAllocator
                  1904640 |            1489920 |         80 | memory/mutexAllocator
                  1105920 |             188544 |         96 | memory/eventAllocator
                  2359296 |             129024 |       1024 | memory/ArenaBlock
      

      this box will crash every 2days, so the memory waste may no that high

      2, our production reverse system:
      physics memory: 16G
      RAM cache: 8G
      DISK: 1516 GB
      average_object_size 16384
      and it run for a much long time:

           allocated      |        in-use      | type size  |   free list name
      --------------------|--------------------|------------|----------------------------------
                805306368 |                  0 |    2097152 | memory/ioBufAllocator[14]
                738197504 |            8388608 |    1048576 | memory/ioBufAllocator[13]
               1258291200 |           46661632 |     524288 | memory/ioBufAllocator[12]
               1300234240 |          183762944 |     262144 | memory/ioBufAllocator[11]
               1170210816 |          466223104 |     131072 | memory/ioBufAllocator[10]
               1790967808 |         1223426048 |      65536 | memory/ioBufAllocator[9]
               2970615808 |         2601418752 |      32768 | memory/ioBufAllocator[8]
               2067791872 |         2044608512 |      16384 | memory/ioBufAllocator[7]
               1169424384 |         1169121280 |       8192 | memory/ioBufAllocator[6]
                711458816 |          710463488 |       4096 | memory/ioBufAllocator[5]
                  1572864 |                  0 |       2048 | memory/ioBufAllocator[4]
                   131072 |                  0 |       1024 | memory/ioBufAllocator[3]
                    65536 |                  0 |        512 | memory/ioBufAllocator[2]
                    32768 |                  0 |        256 | memory/ioBufAllocator[1]
                    16384 |                  0 |        128 | memory/ioBufAllocator[0]
                        0 |                  0 |        576 | memory/ICPRequestCont_allocator
                        0 |                  0 |        112 | memory/ICPPeerReadContAllocator
                        0 |                  0 |        432 | memory/PeerReadDataAllocator
                        0 |                  0 |         32 | memory/MIMEFieldSDKHandle
                        0 |                  0 |        240 | memory/INKVConnAllocator
                        0 |                  0 |         96 | memory/INKContAllocator
                     4096 |                  0 |         32 | memory/apiHookAllocator
                        0 |                  0 |        288 | memory/FetchSMAllocator
                        0 |                  0 |         80 | memory/prefetchLockHandlerAllocator
                        0 |                  0 |        176 | memory/PrefetchBlasterAllocator
                        0 |                  0 |         80 | memory/prefetchUrlBlaster
                        0 |                  0 |         96 | memory/blasterUrlList
                        0 |                  0 |         96 | memory/prefetchUrlEntryAllocator
                        0 |                  0 |        128 | memory/socksProxyAllocator
                        0 |                  0 |        144 | memory/ObjectReloadCont
                  1136640 |             125504 |        592 | memory/httpClientSessionAllocator
                   372736 |              27248 |        208 | memory/httpServerSessionAllocator
                 11317248 |              39296 |       9824 | memory/httpSMAllocator
                        0 |                  0 |         32 | memory/CacheLookupHttpConfigAllocator
                        0 |                  0 |       9888 | memory/httpUpdateSMAllocator
                        0 |                  0 |        128 | memory/RemapPluginsAlloc
                        0 |                  0 |        512 | memory/HCSMAllocator
                        0 |                  0 |         48 | memory/VCEntryAllocator
                        0 |                  0 |         96 | memory/HCEntryAllocator
                        0 |                  0 |         64 | memory/HCHandlerAllocator
                        0 |                  0 |         48 | memory/CongestRequestParamAllocator
                        0 |                  0 |        128 | memory/CongestionDBContAllocator
                  6029312 |             643072 |       2048 | memory/hdrStrHeap
                  7077888 |             657408 |       2048 | memory/hdrHeap
                    26624 |                208 |        208 | memory/httpCacheAltAllocator
                        0 |                  0 |        112 | memory/OneWayTunnelAllocator
                   630784 |               1232 |       1232 | memory/hostDBContAllocator
                   238560 |              17040 |      17040 | memory/dnsBufAllocator
                   161792 |                  0 |       1264 | memory/dnsEntryAllocator
                        0 |                  0 |         16 | memory/DNSRequestDataAllocator
                        0 |                  0 |       1072 | memory/SRVAllocator
                        0 |                  0 |         48 | memory/ClusterVConnectionCache::Entry
                        0 |                  0 |        560 | memory/cacheContAllocator
                        0 |                  0 |        112 | memory/inControlAllocator
                        0 |                  0 |        112 | memory/outControlAllocator
                        0 |                  0 |         32 | memory/byteBankAllocator
                        0 |                  0 |        576 | memory/clusterVCAllocator
                        0 |                  0 |         48 | memory/evacuationKey
                     6144 |                  0 |         48 | memory/cacheRemoveCont
                 17006592 |           14972928 |         96 | memory/evacuationBlock
                  1777664 |             759872 |        992 | memory/cacheVConnection
                   307200 |             111520 |        160 | memory/openDirEntry
                        0 |                  0 |         64 | memory/RamCacheLRUEntry
                104275968 |          104274048 |         96 | memory/RamCacheCLFUSEntry
                  3440640 |            1819200 |        960 | memory/netVCAllocator
                        0 |                  0 |        128 | memory/udpReadContAllocator
                        0 |                  0 |        128 | memory/udpWorkContinuationAllocator
                        0 |                  0 |        160 | memory/udpPacketAllocator
                        0 |                  0 |        304 | memory/socksAllocator
                        0 |                  0 |       1088 | memory/sslNetVCAllocator
                        0 |                  0 |        128 | memory/UDPIOEventAllocator
                   237568 |              22528 |         64 | memory/ioBlockAllocator
                 26087424 |           26081904 |         48 | memory/ioDataAllocator
                   890880 |              84240 |        240 | memory/ioAllocator
                  1525760 |            1403440 |         80 | memory/mutexAllocator
                   565248 |             129696 |         96 | memory/eventAllocator
                  1179648 |               4096 |       1024 | memory/ArenaBlock
      
      

      our team is working on the memory free issue, trying to improve the memory management. and this a big project, the more input|comment the better.

      1. 0004-TS-1006-Make-InkFreeList-memory-pool-configurable.patch
        6 kB
        Yunkai Zhang
      2. 0003-TS-1006-Introduce-a-reclaimable-InkFreeList.patch
        33 kB
        Yunkai Zhang
      3. 0002-TS-1006-Add-a-new-wrapper-ink_atomic_decrement.patch
        1 kB
        Yunkai Zhang
      4. 0001-TS-1006-Add-an-enable-reclaimable-freelist-option.patch
        2 kB
        Yunkai Zhang
      5. Memory-Usage-After-Introduced-New-Allocator.png
        16 kB
        Yunkai Zhang
      6. memusage.ods
        15 kB
        Zhao Yongming
      7. memusage.ods
        15 kB
        Zhao Yongming

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Yunkai Zhang
              Reporter:
              Zhao Yongming
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development