Traffic Server
  1. Traffic Server
  2. TS-1006

memory management, cut down memory waste ?

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.1
    • Fix Version/s: 3.3.4
    • Component/s: Core
    • Labels:
      None

      Description

      when we review the memory usage in the production, there is something abnormal, ie, looks like TS take much memory than index data + common system waste, and here is some memory dump result by set "proxy.config.dump_mem_info_frequency"

      1, the one on a not so busy forwarding system:
      physics memory: 32G
      RAM cache: 22G
      DISK: 6140 GB
      average_object_size 64000

           allocated      |        in-use      | type size  |   free list name
      --------------------|--------------------|------------|----------------------------------
                671088640 |           37748736 |    2097152 | memory/ioBufAllocator[14]
               2248146944 |         2135949312 |    1048576 | memory/ioBufAllocator[13]
               1711276032 |         1705508864 |     524288 | memory/ioBufAllocator[12]
               1669332992 |         1667760128 |     262144 | memory/ioBufAllocator[11]
               2214592512 |         2211840000 |     131072 | memory/ioBufAllocator[10]
               2325741568 |         2323775488 |      65536 | memory/ioBufAllocator[9]
               2091909120 |         2089123840 |      32768 | memory/ioBufAllocator[8]
               1956642816 |         1956478976 |      16384 | memory/ioBufAllocator[7]
               2094530560 |         2094071808 |       8192 | memory/ioBufAllocator[6]
                356515840 |          355540992 |       4096 | memory/ioBufAllocator[5]
                  1048576 |              14336 |       2048 | memory/ioBufAllocator[4]
                   131072 |                  0 |       1024 | memory/ioBufAllocator[3]
                    65536 |                  0 |        512 | memory/ioBufAllocator[2]
                    32768 |                  0 |        256 | memory/ioBufAllocator[1]
                    16384 |                  0 |        128 | memory/ioBufAllocator[0]
                        0 |                  0 |        576 | memory/ICPRequestCont_allocator
                        0 |                  0 |        112 | memory/ICPPeerReadContAllocator
                        0 |                  0 |        432 | memory/PeerReadDataAllocator
                        0 |                  0 |         32 | memory/MIMEFieldSDKHandle
                        0 |                  0 |        240 | memory/INKVConnAllocator
                        0 |                  0 |         96 | memory/INKContAllocator
                     4096 |                  0 |         32 | memory/apiHookAllocator
                        0 |                  0 |        288 | memory/FetchSMAllocator
                        0 |                  0 |         80 | memory/prefetchLockHandlerAllocator
                        0 |                  0 |        176 | memory/PrefetchBlasterAllocator
                        0 |                  0 |         80 | memory/prefetchUrlBlaster
                        0 |                  0 |         96 | memory/blasterUrlList
                        0 |                  0 |         96 | memory/prefetchUrlEntryAllocator
                        0 |                  0 |        128 | memory/socksProxyAllocator
                        0 |                  0 |        144 | memory/ObjectReloadCont
                  3258368 |             576016 |        592 | memory/httpClientSessionAllocator
                   825344 |             139568 |        208 | memory/httpServerSessionAllocator
                 22597632 |            1284848 |       9808 | memory/httpSMAllocator
                        0 |                  0 |         32 | memory/CacheLookupHttpConfigAllocator
                        0 |                  0 |       9856 | memory/httpUpdateSMAllocator
                        0 |                  0 |        128 | memory/RemapPluginsAlloc
                        0 |                  0 |         48 | memory/CongestRequestParamAllocator
                        0 |                  0 |        128 | memory/CongestionDBContAllocator
                  5767168 |             704512 |       2048 | memory/hdrStrHeap
                 18350080 |            1153024 |       2048 | memory/hdrHeap
                    53248 |               2912 |        208 | memory/httpCacheAltAllocator
                        0 |                  0 |        112 | memory/OneWayTunnelAllocator
                   157696 |               1232 |       1232 | memory/hostDBContAllocator
                   102240 |              17040 |      17040 | memory/dnsBufAllocator
                   323584 |                  0 |       1264 | memory/dnsEntryAllocator
                        0 |                  0 |         16 | memory/DNSRequestDataAllocator
                        0 |                  0 |       1072 | memory/SRVAllocator
                        0 |                  0 |         48 | memory/ClusterVConnectionCache::Entry
                        0 |                  0 |        560 | memory/cacheContAllocator
                        0 |                  0 |        112 | memory/inControlAllocator
                        0 |                  0 |        112 | memory/outControlAllocator
                        0 |                  0 |         32 | memory/byteBankAllocator
                        0 |                  0 |        576 | memory/clusterVCAllocator
                        0 |                  0 |         48 | memory/evacuationKey
                     6144 |                  0 |         48 | memory/cacheRemoveCont
                   270336 |             262560 |         96 | memory/evacuationBlock
                  4997120 |            3968416 |        976 | memory/cacheVConnection
                   798720 |             522080 |        160 | memory/openDirEntry
                        0 |                  0 |         64 | memory/RamCacheLRUEntry
                 56426496 |           56426304 |         96 | memory/RamCacheCLFUSEntry
                  9584640 |            6168000 |        960 | memory/netVCAllocator
                        0 |                  0 |        128 | memory/udpReadContAllocator
                        0 |                  0 |        128 | memory/udpWorkContinuationAllocator
                        0 |                  0 |        160 | memory/udpPacketAllocator
                        0 |                  0 |        304 | memory/socksAllocator
                   139264 |              68544 |       1088 | memory/sslNetVCAllocator
                        0 |                  0 |        128 | memory/UDPIOEventAllocator
                   671744 |             115520 |         64 | memory/ioBlockAllocator
                 28305408 |           28301520 |         48 | memory/ioDataAllocator
                  2273280 |             406320 |        240 | memory/ioAllocator
                  1904640 |            1489920 |         80 | memory/mutexAllocator
                  1105920 |             188544 |         96 | memory/eventAllocator
                  2359296 |             129024 |       1024 | memory/ArenaBlock
      

      this box will crash every 2days, so the memory waste may no that high

      2, our production reverse system:
      physics memory: 16G
      RAM cache: 8G
      DISK: 1516 GB
      average_object_size 16384
      and it run for a much long time:

           allocated      |        in-use      | type size  |   free list name
      --------------------|--------------------|------------|----------------------------------
                805306368 |                  0 |    2097152 | memory/ioBufAllocator[14]
                738197504 |            8388608 |    1048576 | memory/ioBufAllocator[13]
               1258291200 |           46661632 |     524288 | memory/ioBufAllocator[12]
               1300234240 |          183762944 |     262144 | memory/ioBufAllocator[11]
               1170210816 |          466223104 |     131072 | memory/ioBufAllocator[10]
               1790967808 |         1223426048 |      65536 | memory/ioBufAllocator[9]
               2970615808 |         2601418752 |      32768 | memory/ioBufAllocator[8]
               2067791872 |         2044608512 |      16384 | memory/ioBufAllocator[7]
               1169424384 |         1169121280 |       8192 | memory/ioBufAllocator[6]
                711458816 |          710463488 |       4096 | memory/ioBufAllocator[5]
                  1572864 |                  0 |       2048 | memory/ioBufAllocator[4]
                   131072 |                  0 |       1024 | memory/ioBufAllocator[3]
                    65536 |                  0 |        512 | memory/ioBufAllocator[2]
                    32768 |                  0 |        256 | memory/ioBufAllocator[1]
                    16384 |                  0 |        128 | memory/ioBufAllocator[0]
                        0 |                  0 |        576 | memory/ICPRequestCont_allocator
                        0 |                  0 |        112 | memory/ICPPeerReadContAllocator
                        0 |                  0 |        432 | memory/PeerReadDataAllocator
                        0 |                  0 |         32 | memory/MIMEFieldSDKHandle
                        0 |                  0 |        240 | memory/INKVConnAllocator
                        0 |                  0 |         96 | memory/INKContAllocator
                     4096 |                  0 |         32 | memory/apiHookAllocator
                        0 |                  0 |        288 | memory/FetchSMAllocator
                        0 |                  0 |         80 | memory/prefetchLockHandlerAllocator
                        0 |                  0 |        176 | memory/PrefetchBlasterAllocator
                        0 |                  0 |         80 | memory/prefetchUrlBlaster
                        0 |                  0 |         96 | memory/blasterUrlList
                        0 |                  0 |         96 | memory/prefetchUrlEntryAllocator
                        0 |                  0 |        128 | memory/socksProxyAllocator
                        0 |                  0 |        144 | memory/ObjectReloadCont
                  1136640 |             125504 |        592 | memory/httpClientSessionAllocator
                   372736 |              27248 |        208 | memory/httpServerSessionAllocator
                 11317248 |              39296 |       9824 | memory/httpSMAllocator
                        0 |                  0 |         32 | memory/CacheLookupHttpConfigAllocator
                        0 |                  0 |       9888 | memory/httpUpdateSMAllocator
                        0 |                  0 |        128 | memory/RemapPluginsAlloc
                        0 |                  0 |        512 | memory/HCSMAllocator
                        0 |                  0 |         48 | memory/VCEntryAllocator
                        0 |                  0 |         96 | memory/HCEntryAllocator
                        0 |                  0 |         64 | memory/HCHandlerAllocator
                        0 |                  0 |         48 | memory/CongestRequestParamAllocator
                        0 |                  0 |        128 | memory/CongestionDBContAllocator
                  6029312 |             643072 |       2048 | memory/hdrStrHeap
                  7077888 |             657408 |       2048 | memory/hdrHeap
                    26624 |                208 |        208 | memory/httpCacheAltAllocator
                        0 |                  0 |        112 | memory/OneWayTunnelAllocator
                   630784 |               1232 |       1232 | memory/hostDBContAllocator
                   238560 |              17040 |      17040 | memory/dnsBufAllocator
                   161792 |                  0 |       1264 | memory/dnsEntryAllocator
                        0 |                  0 |         16 | memory/DNSRequestDataAllocator
                        0 |                  0 |       1072 | memory/SRVAllocator
                        0 |                  0 |         48 | memory/ClusterVConnectionCache::Entry
                        0 |                  0 |        560 | memory/cacheContAllocator
                        0 |                  0 |        112 | memory/inControlAllocator
                        0 |                  0 |        112 | memory/outControlAllocator
                        0 |                  0 |         32 | memory/byteBankAllocator
                        0 |                  0 |        576 | memory/clusterVCAllocator
                        0 |                  0 |         48 | memory/evacuationKey
                     6144 |                  0 |         48 | memory/cacheRemoveCont
                 17006592 |           14972928 |         96 | memory/evacuationBlock
                  1777664 |             759872 |        992 | memory/cacheVConnection
                   307200 |             111520 |        160 | memory/openDirEntry
                        0 |                  0 |         64 | memory/RamCacheLRUEntry
                104275968 |          104274048 |         96 | memory/RamCacheCLFUSEntry
                  3440640 |            1819200 |        960 | memory/netVCAllocator
                        0 |                  0 |        128 | memory/udpReadContAllocator
                        0 |                  0 |        128 | memory/udpWorkContinuationAllocator
                        0 |                  0 |        160 | memory/udpPacketAllocator
                        0 |                  0 |        304 | memory/socksAllocator
                        0 |                  0 |       1088 | memory/sslNetVCAllocator
                        0 |                  0 |        128 | memory/UDPIOEventAllocator
                   237568 |              22528 |         64 | memory/ioBlockAllocator
                 26087424 |           26081904 |         48 | memory/ioDataAllocator
                   890880 |              84240 |        240 | memory/ioAllocator
                  1525760 |            1403440 |         80 | memory/mutexAllocator
                   565248 |             129696 |         96 | memory/eventAllocator
                  1179648 |               4096 |       1024 | memory/ArenaBlock
      
      

      our team is working on the memory free issue, trying to improve the memory management. and this a big project, the more input|comment the better.

      1. 0001-TS-1006-Add-an-enable-reclaimable-freelist-option.patch
        2 kB
        Yunkai Zhang
      2. 0002-TS-1006-Add-a-new-wrapper-ink_atomic_decrement.patch
        1 kB
        Yunkai Zhang
      3. 0003-TS-1006-Introduce-a-reclaimable-InkFreeList.patch
        33 kB
        Yunkai Zhang
      4. 0004-TS-1006-Make-InkFreeList-memory-pool-configurable.patch
        6 kB
        Yunkai Zhang
      5. Memory-Usage-After-Introduced-New-Allocator.png
        16 kB
        Yunkai Zhang
      6. memusage.ods
        15 kB
        Zhao Yongming
      7. memusage.ods
        15 kB
        Zhao Yongming

        Issue Links

          Activity

          Hide
          ASF subversion and git services added a comment -

          Commit 1bb12edf9cd9b0076eaf39b8665909846743ee73 in branch refs/heads/3.3.x from Leif Hedstrom
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1bb12ed ]

          Merge branch 'master' into 3.3.x

          • master: (295 commits)
            TS-2081 Make the WCCP addr configuration LOCAL
            TS-2093: Check bounds on plugin stat creation.
            TS-2092 Use of uninitialized member in HdrHeap.
            TS-2052 ET_SSL thread spinning
            TS-2090 Make proxy.config.allocator.enable_reclaim default based on build instructions
            TS-1006: adjust some reclaimable-freelist's default configuration
            Updated newish tests to the gitignore
            Fix autoconf checks for mcheck_pedantic()
            Fix configure check for eventfd
            TS-1330: Fix logging core at checkout_write()
            add some standard extensions to ignore list
            TS-1953: remove version check from stable plugins
            TS-1953: Remove check_ts_version() from experimental plugins
            TS-1953: remove check_ts_version() from examples
            Added TS-2086.
            TS-2086 Remove a few more unused configs
            Added TS-1685
            TS-1685 Remove TS_MICRO and fellas
            Added Ts-1255
            TS-1255 Fix the types for all overridable configs. This was actually a real bug in the code, in that all float configurations were actually treated as integer, rendering them useless
            ...
          Show
          ASF subversion and git services added a comment - Commit 1bb12edf9cd9b0076eaf39b8665909846743ee73 in branch refs/heads/3.3.x from Leif Hedstrom [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1bb12ed ] Merge branch 'master' into 3.3.x master: (295 commits) TS-2081 Make the WCCP addr configuration LOCAL TS-2093 : Check bounds on plugin stat creation. TS-2092 Use of uninitialized member in HdrHeap. TS-2052 ET_SSL thread spinning TS-2090 Make proxy.config.allocator.enable_reclaim default based on build instructions TS-1006 : adjust some reclaimable-freelist's default configuration Updated newish tests to the gitignore Fix autoconf checks for mcheck_pedantic() Fix configure check for eventfd TS-1330 : Fix logging core at checkout_write() add some standard extensions to ignore list TS-1953 : remove version check from stable plugins TS-1953 : Remove check_ts_version() from experimental plugins TS-1953 : remove check_ts_version() from examples Added TS-2086 . TS-2086 Remove a few more unused configs Added TS-1685 TS-1685 Remove TS_MICRO and fellas Added Ts-1255 TS-1255 Fix the types for all overridable configs. This was actually a real bug in the code, in that all float configurations were actually treated as integer, rendering them useless ...
          Hide
          ASF subversion and git services added a comment -

          Commit cdbfe1d6d6c43241d773a32e3bbdddddd0ca1d9b in branch refs/heads/master from Yunkai Zhang
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=cdbfe1d ]

          TS-1006: adjust some reclaimable-freelist's default configuration

          1) Adjust "enable_reclaim" to "1" as default in
          records.config.default.in file.

          2) Adjust "max_overage" to "3" in code, so that it keeps the same value
          with records.config.default.in. And most important, this value seems
          work well in our product environment.

          Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>

          Show
          ASF subversion and git services added a comment - Commit cdbfe1d6d6c43241d773a32e3bbdddddd0ca1d9b in branch refs/heads/master from Yunkai Zhang [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=cdbfe1d ] TS-1006 : adjust some reclaimable-freelist's default configuration 1) Adjust "enable_reclaim" to "1" as default in records.config.default.in file. 2) Adjust "max_overage" to "3" in code, so that it keeps the same value with records.config.default.in. And most important, this value seems work well in our product environment. Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
          Hide
          Leif Hedstrom added a comment -

          Closing this, since I believe this was landed.

          Show
          Leif Hedstrom added a comment - Closing this, since I believe this was landed.
          Hide
          Leif Hedstrom added a comment -

          Moving out to v3.3.6 for now, which means unless someone moves it back to v3.3.5, this will not go into v3.4.0.

          Show
          Leif Hedstrom added a comment - Moving out to v3.3.6 for now, which means unless someone moves it back to v3.3.5, this will not go into v3.4.0.
          Hide
          Leif Hedstrom added a comment -

          Moving back to v3.3.5 for now.

          Show
          Leif Hedstrom added a comment - Moving back to v3.3.5 for now.
          Hide
          Yunkai Zhang added a comment -

          Do you use master for testing? Or does this issue happen on your private branch which had been backported from master?

          It works well for us in product environment for more than one months.

          Show
          Yunkai Zhang added a comment - Do you use master for testing? Or does this issue happen on your private branch which had been backported from master? It works well for us in product environment for more than one months.
          Hide
          jaekyung oh added a comment -

          Hi. Yunkai Zhang.

          I tried again with "./configure --enable-reclaimable-freelist"

          but same happens...

          these are the part of traffic.out when i executed "traffic_line -x"

          ....
          360192 | 360192 | 672 | memory/netVCAllocator
          0 | 0 | 120 | memory/udpReadContAllocator
          0 | 0 | 176 | memory/udpPacketAllocator
          0 | 0 | 384 | memory/socksAllocator
          0 | 0 | 128 | memory/UDPIOEventAllocator
          16256 | 16256 | 64 | memory/ioBlockAllocator
          8064 | 8064 | 48 | memory/ioDataAllocator
          32480 | 32480 | 232 | memory/ioAllocator
          109512 | 109512 | 72 | memory/mutexAllocator
          318032 | 318032 | 88 | memory/eventAllocator
          268288 | 268288 | 1024 | memory/ArenaBlock
          [Apr 1 18:18:35.010] Manager

          {0x7f25beffd700}

          NOTE: User has changed config file records.config
          [Apr 1 18:18:36.678] Server

          {0x2b989b06b700}

          NOTE: cache enabled
          [2b989b06b700:01][ink_queue_ext.cc:00577][F] 13.28M t:278 f:274 m:0 avg:0.0 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
          [2b989b06b700:01][ink_queue_ext.cc:00584][-] 13.28M t:278 f:274 m:0 avg:0.0 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
          [2b989b06b700:01][ink_queue_ext.cc:00577][F] 13.32M t:278 f:274 m:0 avg:82.2 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
          [2b989b06b700:01][ink_queue_ext.cc:00584][-] 13.32M t:196 f:192 m:0 avg:82.2 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
          NOTE: Traffic Server received Sig 11: Segmentation fault <------------------------------------------------ what happened after logging?
          /usr/local/nts/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0xf2d0)[0x2b9897f1f2d0]
          [0x2b989b274f10]
          [Apr 1 18:18:39.800] Manager

          {0x7f25c7ec0720} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
          [Apr 1 18:18:39.800] Manager {0x7f25c7ec0720}

          ERROR: (last system error 2: No such file or directory)
          [Apr 1 18:18:39.800] Manager

          {0x7f25c7ec0720} ERROR: [Alarms::signalAlarm] Server Process was reset
          [Apr 1 18:18:39.800] Manager {0x7f25c7ec0720}

          ERROR: (last system error 2: No such file or directory)
          [Apr 1 18:18:40.803] Manager

          {0x7f25c7ec0720} NOTE: [LocalManager::startProxy] Launching ts process
          [TrafficServer] using root directory '/usr/local/nts'
          [Apr 1 18:18:40.813] Manager {0x7f25c7ec0720}

          NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '10'
          [Apr 1 18:18:40.813] Manager

          {0x7f25c7ec0720}

          NOTE: [Alarms::signalAlarm] Server Process born
          .....

          Show
          jaekyung oh added a comment - Hi. Yunkai Zhang. I tried again with "./configure --enable-reclaimable-freelist" but same happens... these are the part of traffic.out when i executed "traffic_line -x" .... 360192 | 360192 | 672 | memory/netVCAllocator 0 | 0 | 120 | memory/udpReadContAllocator 0 | 0 | 176 | memory/udpPacketAllocator 0 | 0 | 384 | memory/socksAllocator 0 | 0 | 128 | memory/UDPIOEventAllocator 16256 | 16256 | 64 | memory/ioBlockAllocator 8064 | 8064 | 48 | memory/ioDataAllocator 32480 | 32480 | 232 | memory/ioAllocator 109512 | 109512 | 72 | memory/mutexAllocator 318032 | 318032 | 88 | memory/eventAllocator 268288 | 268288 | 1024 | memory/ArenaBlock [Apr 1 18:18:35.010] Manager {0x7f25beffd700} NOTE: User has changed config file records.config [Apr 1 18:18:36.678] Server {0x2b989b06b700} NOTE: cache enabled [2b989b06b700:01] [ink_queue_ext.cc:00577] [F] 13.28M t:278 f:274 m:0 avg:0.0 M:4 csbase:256 csize:278 tsize:88 cbsize:24576 [2b989b06b700:01] [ink_queue_ext.cc:00584] [-] 13.28M t:278 f:274 m:0 avg:0.0 M:4 csbase:256 csize:278 tsize:88 cbsize:24576 [2b989b06b700:01] [ink_queue_ext.cc:00577] [F] 13.32M t:278 f:274 m:0 avg:82.2 M:4 csbase:256 csize:278 tsize:88 cbsize:24576 [2b989b06b700:01] [ink_queue_ext.cc:00584] [-] 13.32M t:196 f:192 m:0 avg:82.2 M:4 csbase:256 csize:278 tsize:88 cbsize:24576 NOTE: Traffic Server received Sig 11: Segmentation fault <------------------------------------------------ what happened after logging? /usr/local/nts/bin/traffic_server - STACK TRACE: /lib64/libpthread.so.0(+0xf2d0) [0x2b9897f1f2d0] [0x2b989b274f10] [Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault [Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: (last system error 2: No such file or directory) [Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: [Alarms::signalAlarm] Server Process was reset [Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: (last system error 2: No such file or directory) [Apr 1 18:18:40.803] Manager {0x7f25c7ec0720} NOTE: [LocalManager::startProxy] Launching ts process [TrafficServer] using root directory '/usr/local/nts' [Apr 1 18:18:40.813] Manager {0x7f25c7ec0720} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '10' [Apr 1 18:18:40.813] Manager {0x7f25c7ec0720} NOTE: [Alarms::signalAlarm] Server Process born .....
          Hide
          Yunkai Zhang added a comment - - edited

          The format seems error,

          it should be "./configure --enable-reclaimable-freelist", need not "=yes".

          Show
          Yunkai Zhang added a comment - - edited The format seems error, it should be "./configure --enable-reclaimable-freelist", need not "=yes".
          Hide
          jaekyung oh added a comment -

          yes. with last final patch i always put "--enable-reclaimable-freelist=yes"

          same error happens on both 3.2.0 and 3.2.4

          Show
          jaekyung oh added a comment - yes. with last final patch i always put "--enable-reclaimable-freelist=yes" same error happens on both 3.2.0 and 3.2.4
          Hide
          Yunkai Zhang added a comment -

          do you compile ATS with --enable-reclaimable-freelist option?

          Show
          Yunkai Zhang added a comment - do you compile ATS with --enable-reclaimable-freelist option?
          Hide
          jaekyung oh added a comment -

          i've applied final patch on 3.2.0 and 3.2.4. it seems fine without debug options.

          but when i tried turning on CONFIG proxy.config.allocator.debug_filter INT 0 -> 1, no debug message was printed.

          so i tried to turn on CONFIG proxy.config.dump_mem_info_frequency INT 0 -> 1, memory dump message were printed.

          but if i repeat it 3times("0 -> 1 -> traffic_line -x -> 1 -> 0 -> traffic_line -x" is one time) traffic server stops showing these messages.

          [Mar 26 22:16:56.218] Manager

          {0x7f1d6effd700}

          NOTE: User has changed config file records.config
          NOTE: Traffic Server received Sig 11: Segmentation fault
          /usr/local/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0xf2d0)[0x2b59c7a3a2d0]
          [0x2b59cc49d3e0]
          [Mar 26 22:17:02.986] Manager

          {0x7f1d77a93720}

          ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault

          [Mar 26 22:30:10.508] Manager

          {0x7f7d57fff700}

          NOTE: User has changed config file records.config
          NOTE: Traffic Server received Sig 11: Segmentation fault
          /usr/local/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0xfd00)[0x2b3998975d00]
          /usr/local/bin/traffic_server(_Z12init_trackerPKc8RecDataT7RecDataPv+0x31)[0x4eb6f1]
          /usr/local/bin/traffic_server(_Z22RecExecConfigUpdateCbsv+0x79)[0x6a9c99]
          /usr/local/bin/traffic_server(_ZN18config_update_cont14exec_callbacksEiP5Event+0x26)[0x6ab976]
          /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0xb93)[0x6b58e3]
          /usr/local/bin/traffic_server[0x6b3572]
          /lib64/libpthread.so.0(+0x7f05)[0x2b399896df05]
          /lib64/libc.so.6(clone+0x6d)[0x2b399abbe10d]

          Show
          jaekyung oh added a comment - i've applied final patch on 3.2.0 and 3.2.4. it seems fine without debug options. but when i tried turning on CONFIG proxy.config.allocator.debug_filter INT 0 -> 1, no debug message was printed. so i tried to turn on CONFIG proxy.config.dump_mem_info_frequency INT 0 -> 1, memory dump message were printed. but if i repeat it 3times("0 -> 1 -> traffic_line -x -> 1 -> 0 -> traffic_line -x" is one time) traffic server stops showing these messages. [Mar 26 22:16:56.218] Manager {0x7f1d6effd700} NOTE: User has changed config file records.config NOTE: Traffic Server received Sig 11: Segmentation fault /usr/local/bin/traffic_server - STACK TRACE: /lib64/libpthread.so.0(+0xf2d0) [0x2b59c7a3a2d0] [0x2b59cc49d3e0] [Mar 26 22:17:02.986] Manager {0x7f1d77a93720} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault [Mar 26 22:30:10.508] Manager {0x7f7d57fff700} NOTE: User has changed config file records.config NOTE: Traffic Server received Sig 11: Segmentation fault /usr/local/bin/traffic_server - STACK TRACE: /lib64/libpthread.so.0(+0xfd00) [0x2b3998975d00] /usr/local/bin/traffic_server(_Z12init_trackerPKc8RecDataT7RecDataPv+0x31) [0x4eb6f1] /usr/local/bin/traffic_server(_Z22RecExecConfigUpdateCbsv+0x79) [0x6a9c99] /usr/local/bin/traffic_server(_ZN18config_update_cont14exec_callbacksEiP5Event+0x26) [0x6ab976] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0xb93) [0x6b58e3] /usr/local/bin/traffic_server [0x6b3572] /lib64/libpthread.so.0(+0x7f05) [0x2b399896df05] /lib64/libc.so.6(clone+0x6d) [0x2b399abbe10d]
          Hide
          ASF subversion and git services added a comment -

          Commit 616203849743e32d446f09a9089da29116982947 from Zhao Yongming <ming.zym@gmail.com>
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=6162038 ]

          TS-1006: Remove unused InkFreeList->offset

          I have simplified ink_freelist_init API by removing offset parameter,
          but I forgot to remove unused InkFreeList->offset when rebasing code.

          So now, the code using an uninitialized value, it would be a bug. Let's
          clean it completely.

          Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
          Signed-off-by: Zhao Yongming <ming.zym@gmail.com>

          Show
          ASF subversion and git services added a comment - Commit 616203849743e32d446f09a9089da29116982947 from Zhao Yongming <ming.zym@gmail.com> [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=6162038 ] TS-1006 : Remove unused InkFreeList->offset I have simplified ink_freelist_init API by removing offset parameter, but I forgot to remove unused InkFreeList->offset when rebasing code. So now, the code using an uninitialized value, it would be a bug. Let's clean it completely. Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com> Signed-off-by: Zhao Yongming <ming.zym@gmail.com>
          Hide
          ASF subversion and git services added a comment -

          Commit 61d0d2c11de9c7c8fe7822438c82deb2848ad107 from Zhao Yongming <ming.zym@gmail.com>
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=61d0d2c ]

          TS-1006: Fix double-linked list initialization in reclaimable_freelist_new

          This bug broke the double-linked list which would prevent self-thread
          "stealing" memory from other threads. It was found by Weijin when he
          reviewed the code, thans a lot!

          Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
          Signed-off-by: Zhao Yongming <ming.zym@gmail.com>

          Show
          ASF subversion and git services added a comment - Commit 61d0d2c11de9c7c8fe7822438c82deb2848ad107 from Zhao Yongming <ming.zym@gmail.com> [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=61d0d2c ] TS-1006 : Fix double-linked list initialization in reclaimable_freelist_new This bug broke the double-linked list which would prevent self-thread "stealing" memory from other threads. It was found by Weijin when he reviewed the code, thans a lot! Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com> Signed-off-by: Zhao Yongming <ming.zym@gmail.com>
          Hide
          ASF subversion and git services added a comment -

          Commit bb6dd3f79a888e0660d22275aad5684291e152bd from Zhao Yongming <ming.zym@gmail.com>
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=bb6dd3f ]

          TS-1006: Fix assert failure in ink_freelist_new

          This patch fix:
          http://ci.apache.org/builders/tserver-trunk-debug/builds/1172/steps/compile_6/logs/stdio

          it's due to careless rebasing of reclaimable freelist patchset.

          Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com>
          Signed-off-by: Zhao Yongming <ming.zym@gmail.com>

          Show
          ASF subversion and git services added a comment - Commit bb6dd3f79a888e0660d22275aad5684291e152bd from Zhao Yongming <ming.zym@gmail.com> [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=bb6dd3f ] TS-1006 : Fix assert failure in ink_freelist_new This patch fix: http://ci.apache.org/builders/tserver-trunk-debug/builds/1172/steps/compile_6/logs/stdio it's due to careless rebasing of reclaimable freelist patchset. Signed-off-by: Yunkai Zhang <qiushu.zyk@taobao.com> Signed-off-by: Zhao Yongming <ming.zym@gmail.com>
          Hide
          ASF subversion and git services added a comment -

          Commit afe372a6f016202277d25d9e916ba369de860bc6 from Zhao Yongming <ming.zym@gmail.com>
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=afe372a ]

          TS-1006: add CHANGES

          Show
          ASF subversion and git services added a comment - Commit afe372a6f016202277d25d9e916ba369de860bc6 from Zhao Yongming <ming.zym@gmail.com> [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=afe372a ] TS-1006 : add CHANGES
          Hide
          Yunkai Zhang added a comment -

          The 5th version.

          1) split them into more pieces.
          2) using List.h instead of a new list.
          3) avoid the unnecessary whitespace and general fix changes(my vim will clean it automatically before).

          Show
          Yunkai Zhang added a comment - The 5th version. 1) split them into more pieces. 2) using List.h instead of a new list. 3) avoid the unnecessary whitespace and general fix changes(my vim will clean it automatically before).
          Hide
          Igor Galić added a comment -

          +1 to James Peach's proposal.

          Show
          Igor Galić added a comment - +1 to James Peach 's proposal.
          Hide
          James Peach added a comment -

          Some thoughts:

          • we should start work on merging this, because I don't think it will get sufficient review outside of trunk
          • unless there is a performance cost, it should always be compiled in but default to disabled at runtime
          • when merging, break it down into smaller, uncontroversial pieces; for example add the configuration options first, then some infrastructure, then more functional pieces; try to avoid the unnecessary whitespace and general fix changes
          • do we really need another linked list?
          • it would be great to have some unit tests

          Show
          James Peach added a comment - Some thoughts: we should start work on merging this, because I don't think it will get sufficient review outside of trunk unless there is a performance cost, it should always be compiled in but default to disabled at runtime when merging, break it down into smaller, uncontroversial pieces; for example add the configuration options first, then some infrastructure, then more functional pieces; try to avoid the unnecessary whitespace and general fix changes do we really need another linked list? it would be great to have some unit tests
          Hide
          Yunkai Zhang added a comment -

          Now, I merged the previous patches into two patches. The most important
          changes in this pathset is that the orignial freelist memory pool was kept.

          I make the new reclaimable memory pool as alterlative one, and it's disable
          by default, so that it will not distrub most users.

          For more detail, see the commit log of each patch.

          Show
          Yunkai Zhang added a comment - Now, I merged the previous patches into two patches. The most important changes in this pathset is that the orignial freelist memory pool was kept. I make the new reclaimable memory pool as alterlative one, and it's disable by default, so that it will not distrub most users. For more detail, see the commit log of each patch.
          Hide
          Zhao Yongming added a comment -

          ram_cache.algorithm: 0, that is CLFUS. maybe the CLFUS is good with our patches? and 72% is about 11G, that sounds very good to me.

          Show
          Zhao Yongming added a comment - ram_cache.algorithm: 0, that is CLFUS. maybe the CLFUS is good with our patches? and 72% is about 11G, that sounds very good to me.
          Hide
          jaekyung oh added a comment -

          sure.

          physical mem of OS: 16G
          ram_cache.size: 8000000000 (8G)
          reclaim_factor: 0.300000
          max_overage: 3

          ram_cache.algorithm: 0

          thanks.

          Show
          jaekyung oh added a comment - sure. physical mem of OS: 16G ram_cache.size: 8000000000 (8G) reclaim_factor: 0.300000 max_overage: 3 ram_cache.algorithm: 0 thanks.
          Hide
          Yunkai Zhang added a comment - - edited

          jaekyung oh

          That is the first gift for me in 2013.

          Can you show us your configuration to here, maybe it will help other users:

          // physical mem of OS: 16G ?
          ram_cache.size: ?
          reclaim_factor: ?
          max_overage: ?
          
          // which algorithm you use?
          ram_cache.algorithm: ?
          

          Happy New Year.

          Show
          Yunkai Zhang added a comment - - edited jaekyung oh That is the first gift for me in 2013 . Can you show us your configuration to here, maybe it will help other users: // physical mem of OS: 16G ? ram_cache.size: ? reclaim_factor: ? max_overage: ? // which algorithm you use? ram_cache.algorithm: ? Happy New Year.
          Hide
          jaekyung oh added a comment -

          Happy New Year.

          after a week monitoring your last patch shows it's effective. At first, for a couple of days memory usage didn't stop increase but then it keeps going between minimum value of 72% and maximum value of 76%.

          Even though I haven't applied 6th patch traffic server is stable now. Thank you.

          Show
          jaekyung oh added a comment - Happy New Year. after a week monitoring your last patch shows it's effective. At first, for a couple of days memory usage didn't stop increase but then it keeps going between minimum value of 72% and maximum value of 76%. Even though I haven't applied 6th patch traffic server is stable now. Thank you.
          Hide
          Yunkai Zhang added a comment - - edited

          [PATCH 6/6] RamCacheLRU: split LRU queue into multiple queues to reduce memory fragment

          This patch was developed by Bin Chen which has been used in our inner branch of product version. I only do some clear work for it.

          In this patch, the original LRU queue was split into multiple LRU queeus according data size(data->_size_index) of each RamCacheLRUEntry. When memory used by RamCacheLRU exceed max_bytes, ->put() function will remove item from the corresponding LRU-queue in which the data with the same size. This method can improve the memory reusing than before, as a result
          memory fragment will be reduced, it will help us to cut down memory usage.

          Show
          Yunkai Zhang added a comment - - edited [PATCH 6/6] RamCacheLRU: split LRU queue into multiple queues to reduce memory fragment This patch was developed by Bin Chen which has been used in our inner branch of product version. I only do some clear work for it. In this patch, the original LRU queue was split into multiple LRU queeus according data size(data->_size_index) of each RamCacheLRUEntry. When memory used by RamCacheLRU exceed max_bytes, ->put() function will remove item from the corresponding LRU-queue in which the data with the same size. This method can improve the memory reusing than before, as a result memory fragment will be reduced, it will help us to cut down memory usage.
          Hide
          jaekyung oh added a comment -

          got it. i'll talk about ram_cache.size with op team. thank you for your immediate and easily understandable explanations.

          Show
          jaekyung oh added a comment - got it. i'll talk about ram_cache.size with op team. thank you for your immediate and easily understandable explanations.
          Hide
          Yunkai Zhang added a comment - - edited

          jaekyung oh

          I'm not sure if there are memory leak in ClassAllocator, if yes, I'll try to fix it.

          But we can confirm that the original version of InkFreeList won't free memory back to OS. So, let's improve InkFreeList firstly. And we should known that, there is no perfect method to completely avoid memory fragment as long as we allocate a big chunk and split it into small type_size. What we could do is to reduce, not to avoid it.

          Of couse, memory fragment will consume some of memory, but should not be too big. Now, we have found that RamCacheLRU may lead to memory fragment, that's Bin Chen's patch to fix.

          So, I have two suggestions:
          1) set a reasonable ram_cache.size and max_overage (we should leave some space for memory fragment). If there are only memory fragment and no memory leak, the total size should not increase infinitely. But in my experience, the total size may increase for several days before it reach the top of memory fragment.
          2) wait for Bin Chen's patch, I'll send it soon, which will reduce memory fragment caused by RamCacheLRU(CLFUS has the same problem but we haven't fix it now, so I suggest you select LRU to test it).

          Show
          Yunkai Zhang added a comment - - edited jaekyung oh I'm not sure if there are memory leak in ClassAllocator, if yes, I'll try to fix it. But we can confirm that the original version of InkFreeList won't free memory back to OS. So, let's improve InkFreeList firstly. And we should known that, there is no perfect method to completely avoid memory fragment as long as we allocate a big chunk and split it into small type_size. What we could do is to reduce, not to avoid it. Of couse, memory fragment will consume some of memory, but should not be too big. Now, we have found that RamCacheLRU may lead to memory fragment, that's Bin Chen 's patch to fix. So, I have two suggestions: 1) set a reasonable ram_cache.size and max_overage (we should leave some space for memory fragment). If there are only memory fragment and no memory leak, the total size should not increase infinitely. But in my experience, the total size may increase for several days before it reach the top of memory fragment. 2) wait for Bin Chen 's patch, I'll send it soon, which will reduce memory fragment caused by RamCacheLRU(CLFUS has the same problem but we haven't fix it now, so I suggest you select LRU to test it).
          Hide
          jaekyung oh added a comment -

          Thank you for your quick feedback.

          Let me ask you a question about Memory used by ClassAllocator to make clear what problem is.

          Of course the total memory will be over 8G in my config but it seems that the memory used by ClassAllocator never returned to OS...

          doesn't it mean memory leak in ClassAllocator?

          in my humble opinion if the memory usage by traffic server can be keep with constant size instead of constantly increasing, it will really better. my op team doesn't like restarting traffic server at regular intervals. and i'm afraid they think traffic server is not reliable in memory control.

          Thanks a lot.

          Show
          jaekyung oh added a comment - Thank you for your quick feedback. Let me ask you a question about Memory used by ClassAllocator to make clear what problem is. Of course the total memory will be over 8G in my config but it seems that the memory used by ClassAllocator never returned to OS... doesn't it mean memory leak in ClassAllocator? in my humble opinion if the memory usage by traffic server can be keep with constant size instead of constantly increasing, it will really better. my op team doesn't like restarting traffic server at regular intervals. and i'm afraid they think traffic server is not reliable in memory control. Thanks a lot.
          Hide
          Yunkai Zhang added a comment - - edited

          HI jaekyung oh

          The total memory used by InkFreeList in the debug log is 8710.2M, it including two parts:
          1) Memory used by RamCache (about 8G in your case).
          2) Memory used by ClassAllocator.

          So, it's not surprise that the total memory usage > 8G.

          I think you might set a smaller ram_cache.size if you want to control the total memory not beyond 9G.


          In addition, there are some optimizations about RamCacheLRU we can do. My coworker, Bin Chen, have developed a patch to fasten the memory used by RamCacheLRU, I'll do some clear work before sent it to here.

          plus Bin Chen's patch, these patch series will be more effective.

          Show
          Yunkai Zhang added a comment - - edited HI jaekyung oh The total memory used by InkFreeList in the debug log is 8710.2M, it including two parts: 1) Memory used by RamCache (about 8G in your case). 2) Memory used by ClassAllocator. So, it's not surprise that the total memory usage > 8G. I think you might set a smaller ram_cache.size if you want to control the total memory not beyond 9G. – In addition, there are some optimizations about RamCacheLRU we can do. My coworker, Bin Chen , have developed a patch to fasten the memory used by RamCacheLRU, I'll do some clear work before sent it to here. plus Bin Chen 's patch, these patch series will be more effective.
          Hide
          jaekyung oh added a comment -

          Hi Yunkai Zhang.

          i'm afraid there is memory leak a little because the memory usage tends to increase. Now traffic server uses 8.7G bytes. Maybe it will use 9G tomorrow.

          we set ram_cache.size 8000000000, ram_cache_cutoff 6000000000

          part of yesterday debug logs :

          [2aac3c888700:40][ink_queue.cc:00616][F] 8599.70M t:12861 f:71 m:71 avg:70.9 M:12790 csbase:64 csize:64 tsize:4096 cbsize:266240
          [2aac3c888700:40][ink_queue.cc:00623][-] 8599.70M t:12791 f:1 m:71 avg:70.9 M:12790 csbase:64 csize:64 tsize:4096 cbsize:266240
          [2aac3c888700:41][ink_queue.cc:00631][M] 8599.70M t:15469 f:0 m:1 avg:6.3 M:15469 csbase:32 csize:32 tsize:8192 cbsize:266240
          [2aac3c888700:41][ink_queue.cc:00634][+] 8599.70M t:15501 f:31 m:1 avg:6.3 M:15469 csbase:32 csize:32 tsize:8192 cbsize:266240
          [2aac3c888700:01][ink_queue.cc:00631][M] 8599.70M t:117925 f:0 m:12 avg:45.7 M:117925 csbase:256 csize:278 tsize:88 cbsize:24576
          [2aac3c888700:01][ink_queue.cc:00634][+] 8599.70M t:118203 f:277 m:12 avg:45.7 M:117925 csbase:256 csize:278 tsize:88 cbsize:24576
          [2aac3c888700:40][ink_queue.cc:00631][M] 8599.70M t:12791 f:0 m:1 avg:9.1 M:12791 csbase:64 csize:64 tsize:4096 cbsize:266240
          [2aac3c888700:40][ink_queue.cc:00634][+] 8599.70M t:12855 f:63 m:1 avg:9.1 M:12791 csbase:64 csize:64 tsize:4096 cbsize:266240
          [2aac3c888700:42][ink_queue.cc:00616][F] 8599.70M t:22873 f:18 m:16 avg:16.1 M:22855 csbase:32 csize:32 tsize:16384 cbsize:528384
          [2aac3c888700:42][ink_queue.cc:00623][-] 8599.70M t:22857 f:2 m:16 avg:16.1 M:22855 csbase:32 csize:32 tsize:16384 cbsize:528384
          [2aac3c888700:43][ink_queue.cc:00616][F] 8599.70M t:4474 f:37 m:26 avg:31.2 M:4437 csbase:32 csize:31 tsize:32768 cbsize:1019904
          [2aac3c888700:43][ink_queue.cc:00623][-] 8599.70M t:4443 f:6 m:26 avg:31.2 M:4437 csbase:32 csize:31 tsize:32768 cbsize:1019904
          [2aac3c888700:47][ink_queue.cc:00616][F] 8599.70M t:264 f:2 m:1 avg:1.3 M:262 csbase:32 csize:1 tsize:524288 cbsize:528384
          [2aac3c888700:47][ink_queue.cc:00623][-] 8599.19M t:263 f:1 m:1 avg:1.3 M:262 csbase:32 csize:1 tsize:524288 cbsize:528384
          [2aac3c888700:46][ink_queue.cc:00631][M] 8599.19M t:733 f:0 m:0 avg:1.1 M:733 csbase:32 csize:3 tsize:262144 cbsize:790528
          [2aac3c888700:46][ink_queue.cc:00634][+] 8599.19M t:736 f:2 m:0 avg:1.1 M:733 csbase:32 csize:3 tsize:262144 cbsize:790528
          [2aac3c888700:27][ink_queue.cc:00616][F] 8599.19M t:157 f:157 m:152 avg:150.7 M:0 csbase:128 csize:129 tsize:2048 cbsize:266240
          [2aac3c888700:27][ink_queue.cc:00623][-] 8598.94M t:7 f:7 m:152 avg:150.7 M:0 csbase:128 csize:129 tsize:2048 cbsize:266240
          [2aac3c888700:41][ink_queue.cc:00616][F] 8598.94M t:15501 f:30 m:30 avg:29.8 M:15471 csbase:32 csize:32 tsize:8192 cbsize:266240

          current last debug logs :

          [2aac3c989700:44][ink_queue.cc:00616][F] 8710.25M t:1461 f:2 m:1 avg:1.3 M:1459 csbase:32 csize:15 tsize:65536 cbsize:987136
          [2aac3c989700:44][ink_queue.cc:00623][-] 8710.25M t:1460 f:1 m:1 avg:1.3 M:1459 csbase:32 csize:15 tsize:65536 cbsize:987136
          [2aac3c989700:45][ink_queue.cc:00616][F] 8710.25M t:1536 f:2 m:1 avg:0.8 M:1534 csbase:32 csize:7 tsize:131072 cbsize:921600
          [2aac3c989700:45][ink_queue.cc:00623][-] 8710.25M t:1536 f:2 m:1 avg:0.8 M:1534 csbase:32 csize:7 tsize:131072 cbsize:921600
          [2aac3c989700:47][ink_queue.cc:00616][F] 8710.25M t:93 f:3 m:2 avg:1.9 M:90 csbase:32 csize:1 tsize:524288 cbsize:528384
          [2aac3c989700:47][ink_queue.cc:00623][-] 8709.75M t:92 f:2 m:2 avg:1.9 M:90 csbase:32 csize:1 tsize:524288 cbsize:528384
          [2aac3c989700:41][ink_queue.cc:00631][M] 8709.75M t:18766 f:0 m:1 avg:0.3 M:18766 csbase:32 csize:32 tsize:8192 cbsize:266240
          [2aac3c989700:41][ink_queue.cc:00634][+] 8710.00M t:18798 f:31 m:1 avg:0.3 M:18766 csbase:32 csize:32 tsize:8192 cbsize:266240
          [2aac3c686700:42][ink_queue.cc:00616][F] 8710.00M t:26868 f:32 m:32 avg:30.6 M:26836 csbase:32 csize:32 tsize:16384 cbsize:528384
          [2aac3c686700:42][ink_queue.cc:00623][-] 8710.00M t:26838 f:2 m:32 avg:30.6 M:26836 csbase:32 csize:32 tsize:16384 cbsize:528384
          [2aac3c686700:01][ink_queue.cc:00631][M] 8710.00M t:141808 f:0 m:2 avg:6.2 M:141808 csbase:256 csize:278 tsize:88 cbsize:24576
          [2aac3c686700:01][ink_queue.cc:00634][+] 8710.02M t:142086 f:277 m:2 avg:6.2 M:141808 csbase:256 csize:278 tsize:88 cbsize:24576
          [2aac3ca8a700:42][ink_queue.cc:00631][M] 8710.02M t:27525 f:0 m:1 avg:0.4 M:27525 csbase:32 csize:32 tsize:16384 cbsize:528384
          [2aac3ca8a700:42][ink_queue.cc:00634][+] 8710.02M t:27557 f:31 m:1 avg:0.4 M:27525 csbase:32 csize:32 tsize:16384 cbsize:528384
          [2aac3c686700:41][ink_queue.cc:00616][F] 8710.02M t:18884 f:32 m:30 avg:30.1 M:18852 csbase:32 csize:32 tsize:8192 cbsize:266240
          [2aac3c686700:41][ink_queue.cc:00623][-] 8710.02M t:18854 f:2 m:30 avg:30.1 M:18852 csbase:32 csize:32 tsize:8192 cbsize:266240
          [2aac3c686700:42][ink_queue.cc:00631][M] 8710.02M t:26838 f:0 m:1 avg:1.8 M:26838 csbase:32 csize:32 tsize:16384 cbsize:528384
          [2aac3c686700:42][ink_queue.cc:00634][+] 8710.02M t:26870 f:31 m:1 avg:1.8 M:26838 csbase:32 csize:32 tsize:16384 cbsize:528384

          if traffic server re-use freed memory(including gather fragment into chunk) (i think) it should keep memory usage around 8G. is there any factor affect memory usage in config?

          thanks and happy new year to everyone.

          Show
          jaekyung oh added a comment - Hi Yunkai Zhang. i'm afraid there is memory leak a little because the memory usage tends to increase. Now traffic server uses 8.7G bytes. Maybe it will use 9G tomorrow. we set ram_cache.size 8000000000, ram_cache_cutoff 6000000000 part of yesterday debug logs : [2aac3c888700:40] [ink_queue.cc:00616] [F] 8599.70M t:12861 f:71 m:71 avg:70.9 M:12790 csbase:64 csize:64 tsize:4096 cbsize:266240 [2aac3c888700:40] [ink_queue.cc:00623] [-] 8599.70M t:12791 f:1 m:71 avg:70.9 M:12790 csbase:64 csize:64 tsize:4096 cbsize:266240 [2aac3c888700:41] [ink_queue.cc:00631] [M] 8599.70M t:15469 f:0 m:1 avg:6.3 M:15469 csbase:32 csize:32 tsize:8192 cbsize:266240 [2aac3c888700:41] [ink_queue.cc:00634] [+] 8599.70M t:15501 f:31 m:1 avg:6.3 M:15469 csbase:32 csize:32 tsize:8192 cbsize:266240 [2aac3c888700:01] [ink_queue.cc:00631] [M] 8599.70M t:117925 f:0 m:12 avg:45.7 M:117925 csbase:256 csize:278 tsize:88 cbsize:24576 [2aac3c888700:01] [ink_queue.cc:00634] [+] 8599.70M t:118203 f:277 m:12 avg:45.7 M:117925 csbase:256 csize:278 tsize:88 cbsize:24576 [2aac3c888700:40] [ink_queue.cc:00631] [M] 8599.70M t:12791 f:0 m:1 avg:9.1 M:12791 csbase:64 csize:64 tsize:4096 cbsize:266240 [2aac3c888700:40] [ink_queue.cc:00634] [+] 8599.70M t:12855 f:63 m:1 avg:9.1 M:12791 csbase:64 csize:64 tsize:4096 cbsize:266240 [2aac3c888700:42] [ink_queue.cc:00616] [F] 8599.70M t:22873 f:18 m:16 avg:16.1 M:22855 csbase:32 csize:32 tsize:16384 cbsize:528384 [2aac3c888700:42] [ink_queue.cc:00623] [-] 8599.70M t:22857 f:2 m:16 avg:16.1 M:22855 csbase:32 csize:32 tsize:16384 cbsize:528384 [2aac3c888700:43] [ink_queue.cc:00616] [F] 8599.70M t:4474 f:37 m:26 avg:31.2 M:4437 csbase:32 csize:31 tsize:32768 cbsize:1019904 [2aac3c888700:43] [ink_queue.cc:00623] [-] 8599.70M t:4443 f:6 m:26 avg:31.2 M:4437 csbase:32 csize:31 tsize:32768 cbsize:1019904 [2aac3c888700:47] [ink_queue.cc:00616] [F] 8599.70M t:264 f:2 m:1 avg:1.3 M:262 csbase:32 csize:1 tsize:524288 cbsize:528384 [2aac3c888700:47] [ink_queue.cc:00623] [-] 8599.19M t:263 f:1 m:1 avg:1.3 M:262 csbase:32 csize:1 tsize:524288 cbsize:528384 [2aac3c888700:46] [ink_queue.cc:00631] [M] 8599.19M t:733 f:0 m:0 avg:1.1 M:733 csbase:32 csize:3 tsize:262144 cbsize:790528 [2aac3c888700:46] [ink_queue.cc:00634] [+] 8599.19M t:736 f:2 m:0 avg:1.1 M:733 csbase:32 csize:3 tsize:262144 cbsize:790528 [2aac3c888700:27] [ink_queue.cc:00616] [F] 8599.19M t:157 f:157 m:152 avg:150.7 M:0 csbase:128 csize:129 tsize:2048 cbsize:266240 [2aac3c888700:27] [ink_queue.cc:00623] [-] 8598.94M t:7 f:7 m:152 avg:150.7 M:0 csbase:128 csize:129 tsize:2048 cbsize:266240 [2aac3c888700:41] [ink_queue.cc:00616] [F] 8598.94M t:15501 f:30 m:30 avg:29.8 M:15471 csbase:32 csize:32 tsize:8192 cbsize:266240 current last debug logs : [2aac3c989700:44] [ink_queue.cc:00616] [F] 8710.25M t:1461 f:2 m:1 avg:1.3 M:1459 csbase:32 csize:15 tsize:65536 cbsize:987136 [2aac3c989700:44] [ink_queue.cc:00623] [-] 8710.25M t:1460 f:1 m:1 avg:1.3 M:1459 csbase:32 csize:15 tsize:65536 cbsize:987136 [2aac3c989700:45] [ink_queue.cc:00616] [F] 8710.25M t:1536 f:2 m:1 avg:0.8 M:1534 csbase:32 csize:7 tsize:131072 cbsize:921600 [2aac3c989700:45] [ink_queue.cc:00623] [-] 8710.25M t:1536 f:2 m:1 avg:0.8 M:1534 csbase:32 csize:7 tsize:131072 cbsize:921600 [2aac3c989700:47] [ink_queue.cc:00616] [F] 8710.25M t:93 f:3 m:2 avg:1.9 M:90 csbase:32 csize:1 tsize:524288 cbsize:528384 [2aac3c989700:47] [ink_queue.cc:00623] [-] 8709.75M t:92 f:2 m:2 avg:1.9 M:90 csbase:32 csize:1 tsize:524288 cbsize:528384 [2aac3c989700:41] [ink_queue.cc:00631] [M] 8709.75M t:18766 f:0 m:1 avg:0.3 M:18766 csbase:32 csize:32 tsize:8192 cbsize:266240 [2aac3c989700:41] [ink_queue.cc:00634] [+] 8710.00M t:18798 f:31 m:1 avg:0.3 M:18766 csbase:32 csize:32 tsize:8192 cbsize:266240 [2aac3c686700:42] [ink_queue.cc:00616] [F] 8710.00M t:26868 f:32 m:32 avg:30.6 M:26836 csbase:32 csize:32 tsize:16384 cbsize:528384 [2aac3c686700:42] [ink_queue.cc:00623] [-] 8710.00M t:26838 f:2 m:32 avg:30.6 M:26836 csbase:32 csize:32 tsize:16384 cbsize:528384 [2aac3c686700:01] [ink_queue.cc:00631] [M] 8710.00M t:141808 f:0 m:2 avg:6.2 M:141808 csbase:256 csize:278 tsize:88 cbsize:24576 [2aac3c686700:01] [ink_queue.cc:00634] [+] 8710.02M t:142086 f:277 m:2 avg:6.2 M:141808 csbase:256 csize:278 tsize:88 cbsize:24576 [2aac3ca8a700:42] [ink_queue.cc:00631] [M] 8710.02M t:27525 f:0 m:1 avg:0.4 M:27525 csbase:32 csize:32 tsize:16384 cbsize:528384 [2aac3ca8a700:42] [ink_queue.cc:00634] [+] 8710.02M t:27557 f:31 m:1 avg:0.4 M:27525 csbase:32 csize:32 tsize:16384 cbsize:528384 [2aac3c686700:41] [ink_queue.cc:00616] [F] 8710.02M t:18884 f:32 m:30 avg:30.1 M:18852 csbase:32 csize:32 tsize:8192 cbsize:266240 [2aac3c686700:41] [ink_queue.cc:00623] [-] 8710.02M t:18854 f:2 m:30 avg:30.1 M:18852 csbase:32 csize:32 tsize:8192 cbsize:266240 [2aac3c686700:42] [ink_queue.cc:00631] [M] 8710.02M t:26838 f:0 m:1 avg:1.8 M:26838 csbase:32 csize:32 tsize:16384 cbsize:528384 [2aac3c686700:42] [ink_queue.cc:00634] [+] 8710.02M t:26870 f:31 m:1 avg:1.8 M:26838 csbase:32 csize:32 tsize:16384 cbsize:528384 if traffic server re-use freed memory(including gather fragment into chunk) (i think) it should keep memory usage around 8G. is there any factor affect memory usage in config? thanks and happy new year to everyone.
          Hide
          Yunkai Zhang added a comment - - edited

          HI jaekyung oh:

          • Do you turn off the debug info(0005 patch enable it by default)? Setting debug_filter to be zero can disable it.
          • Yes, we should adjust max_overage according our environment's character. You can using 'traffic_linx -x' to refresh the new value of max_overage/reclaim_factor/debug_filter options without restart TS. In my experience, 10 maybe too large, I suggest you enlarge max_overage from 3,4,5,.. step by step and find the appropriate value for you.
          • We should compare CPU USAGE with the same QPS. In my testing, these patches may enlarge the QPS at some time(that is a good thing), because it use thread-local cache.
          Show
          Yunkai Zhang added a comment - - edited HI jaekyung oh : Do you turn off the debug info(0005 patch enable it by default)? Setting debug_filter to be zero can disable it. Yes, we should adjust max_overage according our environment's character. You can using 'traffic_linx -x' to refresh the new value of max_overage/reclaim_factor/debug_filter options without restart TS. In my experience, 10 maybe too large, I suggest you enlarge max_overage from 3,4,5,.. step by step and find the appropriate value for you. We should compare CPU USAGE with the same QPS. In my testing, these patches may enlarge the QPS at some time(that is a good thing ), because it use thread-local cache.
          Hide
          jaekyung oh added a comment -

          By the way op team reported the cpu usage got a little higher.

          i think that's because of max_overage is too small. is it right? then what would be proper value? 10 ?

          thank you again Yunkai Zhang.

          Show
          jaekyung oh added a comment - By the way op team reported the cpu usage got a little higher. i think that's because of max_overage is too small. is it right? then what would be proper value? 10 ? thank you again Yunkai Zhang.
          Hide
          jaekyung oh added a comment -

          Hi Yunkai Zhang!

          it's great!! it' been only one day since i applied your last patch. it's very hopeful.

          First the memory usage had been increasing up before it reached 8G.

          Since that time traffic server keeps memory usage around 8G for 12 hours. That's what we want! Thank you so much. Great job!!

          Show
          jaekyung oh added a comment - Hi Yunkai Zhang! it's great!! it' been only one day since i applied your last patch. it's very hopeful. First the memory usage had been increasing up before it reached 8G. Since that time traffic server keeps memory usage around 8G for 12 hours. That's what we want! Thank you so much. Great job!!
          Hide
          bettydramit added a comment -

          HI! Yunkai Zhang
          the same error
          This patch of 0005 to be dependent on the other(0001---00004)?

          Show
          bettydramit added a comment - HI! Yunkai Zhang the same error This patch of 0005 to be dependent on the other(0001---00004)?
          Hide
          Yunkai Zhang added a comment -

          HI bettydramit:

          I have updated the patch.

          Show
          Yunkai Zhang added a comment - HI bettydramit : I have updated the patch.
          Hide
          bettydramit added a comment -

          patch error with 0005-xxx
          + /usr/bin/patch -s -p1 --fuzz=0
          1 out of 7 hunks FAILED – saving rejects to file lib/ts/ink_queue.cc.rej

          Show
          bettydramit added a comment - patch error with 0005-xxx + /usr/bin/patch -s -p1 --fuzz=0 1 out of 7 hunks FAILED – saving rejects to file lib/ts/ink_queue.cc.rej
          Hide
          Yunkai Zhang added a comment -

          only rename this patch's name to '0005-xxx',so that it'll not confuse other users.

          Show
          Yunkai Zhang added a comment - only rename this patch's name to '0005-xxx',so that it'll not confuse other users.
          Hide
          jaekyung oh added a comment - - edited

          thank you for very fast patch.

          i'll try new patch with these config values proxy.config.allocator.reclaim_factor as 0.30000 and proxy.config.allocator.max_overage as 3.

          i'll let you know the result. thank you again.

          Show
          jaekyung oh added a comment - - edited thank you for very fast patch. i'll try new patch with these config values proxy.config.allocator.reclaim_factor as 0.30000 and proxy.config.allocator.max_overage as 3. i'll let you know the result. thank you again.
          Hide
          Yunkai Zhang added a comment - - edited

          HI @jaekyung oh

          Please try the following patch(apply it directly upon other patches):

          Allocator: adjust reclaiming strategy of InkFreeList

          1) Only do recliaming when the code try to allocate memory
          from partial-free Chunks or OS.
          2) Try to reclaim memory from all caches in the same thread,
          this can make the reclaiming more effective.

          Show
          Yunkai Zhang added a comment - - edited HI @ jaekyung oh Please try the following patch(apply it directly upon other patches): Allocator: adjust reclaiming strategy of InkFreeList 1) Only do recliaming when the code try to allocate memory from partial-free Chunks or OS. 2) Try to reclaim memory from all caches in the same thread, this can make the reclaiming more effective.
          Hide
          Yunkai Zhang added a comment -

          Hi jaekyung oh

          Yes, your can decrease max_overage and enlarge reclaim_factor to speed up reclaiming rate.

          But I'm cooking a new patch to adjust reclaiming strategy, which I think will be more effective than before (thanks for your feedback, which help me a lot on optimization).

          So I think, you do not need to try it again, just waiting for my new patch

          Show
          Yunkai Zhang added a comment - Hi jaekyung oh Yes, your can decrease max_overage and enlarge reclaim_factor to speed up reclaiming rate. But I'm cooking a new patch to adjust reclaiming strategy, which I think will be more effective than before (thanks for your feedback, which help me a lot on optimization). So I think, you do not need to try it again, just waiting for my new patch
          Hide
          jaekyung oh added a comment -

          i don't know what value will be adequate. The value was

          CONFIG proxy.config.allocator.enable_reclaim INT 1
          CONFIG proxy.config.allocator.reclaim_factor FLOAT 0.200000
          CONFIG proxy.config.allocator.max_overage INT 100

          if i should adjust these values like below is it ok?
          CONFIG proxy.config.allocator.reclaim_factor FLOAT 0.500000
          CONFIG proxy.config.allocator.max_overage INT 10

          Show
          jaekyung oh added a comment - i don't know what value will be adequate. The value was CONFIG proxy.config.allocator.enable_reclaim INT 1 CONFIG proxy.config.allocator.reclaim_factor FLOAT 0.200000 CONFIG proxy.config.allocator.max_overage INT 100 if i should adjust these values like below is it ok? CONFIG proxy.config.allocator.reclaim_factor FLOAT 0.500000 CONFIG proxy.config.allocator.max_overage INT 10
          Hide
          Yunkai Zhang added a comment -

          bettydramit

          Please read this comment: https://issues.apache.org/jira/browse/TS-1006?focusedCommentId=13539187&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13539187

          If the description is not clear to understand, tell us.

          For example, you can decrease max_overage to speed up reclaiming rate(but may harm performance if max_overage is too small, you can test it).

          Show
          Yunkai Zhang added a comment - bettydramit Please read this comment: https://issues.apache.org/jira/browse/TS-1006?focusedCommentId=13539187&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13539187 If the description is not clear to understand, tell us. For example, you can decrease max_overage to speed up reclaiming rate(but may harm performance if max_overage is too small, you can test it).
          Hide
          bettydramit added a comment -

          How reasonable to set these values​​?

          Show
          bettydramit added a comment - How reasonable to set these values​​?
          Hide
          Yunkai Zhang added a comment -

          jaekyung oh

          1) The reclaiming rate depends on configuration, you can try to adjust: max_overage/reclaim_factor to see the reclaiming effect.

          2) These patches are being improved, it maybe not enough mature.

          Right, it's not easy work, but I have confidence to find the best solution.

          I'm cooking a new patch to adjust the reclaiming strategy, please apply it and test it, I need your help, thank you

          The newest patch will be give soon ...

          Show
          Yunkai Zhang added a comment - jaekyung oh 1) The reclaiming rate depends on configuration, you can try to adjust: max_overage/reclaim_factor to see the reclaiming effect. 2) These patches are being improved, it maybe not enough mature. Right, it's not easy work, but I have confidence to find the best solution . I'm cooking a new patch to adjust the reclaiming strategy, please apply it and test it, I need your help, thank you The newest patch will be give soon ...
          Hide
          jaekyung oh added a comment - - edited

          your patches looks fine but regretfully it doesn't look fundamental solution.

          because the memory usage still keep increasing every moment and finally we couldn't help but restart traffic server today because memory usage reached 90% of 16G.

          in spite of your new patches it seems we have to restart traffic server every 1~2 days to avoid system crash might be caused by run out of memory. it's same situation before your patches....

          Thansk you again.

          Show
          jaekyung oh added a comment - - edited your patches looks fine but regretfully it doesn't look fundamental solution. because the memory usage still keep increasing every moment and finally we couldn't help but restart traffic server today because memory usage reached 90% of 16G. in spite of your new patches it seems we have to restart traffic server every 1~2 days to avoid system crash might be caused by run out of memory. it's same situation before your patches.... Thansk you again.
          Hide
          Yunkai Zhang added a comment -

          jaekyung oh

          Ok, Thanks your feedback

          Show
          Yunkai Zhang added a comment - jaekyung oh Ok, Thanks your feedback
          Hide
          jaekyung oh added a comment -

          i've applied your new patches this morning and no problem yet. it passed 4G where the error happened last time. it looks fine now. i'll let you know if something else happens.

          Show
          jaekyung oh added a comment - i've applied your new patches this morning and no problem yet. it passed 4G where the error happened last time. it looks fine now. i'll let you know if something else happens.
          Hide
          Yunkai Zhang added a comment -

          Limit the size of each chunk and resize alignment, which can decrease
          the aligned memory size when call mmap() to allocate, so it can help
          us to avoid mmap failure.

          After apply this patch, the max alignment should not larger than
          MAX_CHUNK_BYTE_SIZE defined in code.

          Show
          Yunkai Zhang added a comment - Limit the size of each chunk and resize alignment, which can decrease the aligned memory size when call mmap() to allocate, so it can help us to avoid mmap failure. After apply this patch, the max alignment should not larger than MAX_CHUNK_BYTE_SIZE defined in code.
          Hide
          Yunkai Zhang added a comment -

          jaekyung oh

          I'm cooking another new patch which should fix the root cause of this issue, waiting for my good news

          Show
          Yunkai Zhang added a comment - jaekyung oh I'm cooking another new patch which should fix the root cause of this issue, waiting for my good news
          Hide
          Yunkai Zhang added a comment -

          jaekyung oh

          Please try this patch: 0003-Allocator-store-InkChunkInfo-into-Chunk.patch, and give us some feedback.

          Just apply it upon 0001-0002 patches.

          Show
          Yunkai Zhang added a comment - jaekyung oh Please try this patch: 0003-Allocator-store-InkChunkInfo-into-Chunk.patch, and give us some feedback. Just apply it upon 0001-0002 patches.
          Hide
          Yunkai Zhang added a comment -

          Instead of allocating InkChunkInfo by ats_memalign(), store it
          into chunk, it can reduce memory fragment a bit.

          Show
          Yunkai Zhang added a comment - Instead of allocating InkChunkInfo by ats_memalign(), store it into chunk, it can reduce memory fragment a bit.
          Hide
          Yunkai Zhang added a comment -

          Ok, thanks.

          I'll give a patch later, It'll do some optimization - Merge a data struct(InkChunkInfo) into chunk to reduce memory fragment. I'm not sure it will fix this issue, but worth trying.

          Show
          Yunkai Zhang added a comment - Ok, thanks. I'll give a patch later, It'll do some optimization - Merge a data struct(InkChunkInfo) into chunk to reduce memory fragment. I'm not sure it will fix this issue, but worth trying.
          Hide
          jaekyung oh added a comment -

          i've tried again in accordance with your guide.

          [2ba8ae8cc700:43][ink_queue.cc:00601][-] all:4011MB t:1024 f:8 m:40 avg:40.0 malloc:1016 csize:32 tsize:32768 cbsize:1052672
          [2ba8adbb7e20:40][ink_queue.cc:00595][M] all:4029MB t:767 f:87 m:86 avg:84.8 malloc:680 csize:64 tsize:4096 cbsize:266240
          [2ba8adbb7e20:40][ink_queue.cc:00601][-] all:4029MB t:703 f:23 m:86 avg:84.8 malloc:680 csize:64 tsize:4096 cbsize:266240
          [2ba8ae8cc700:03][ink_queue.cc:00668][F] all:4044MB t:628 f:108 m:108 avg:107.1 malloc:519 csize:70 tsize:232 cbsize:16384
          [2ba8ae8cc700:03][ink_queue.cc:00674][-] all:4044MB t:557 f:38 m:108 avg:107.1 malloc:519 csize:70 tsize:232 cbsize:16384
          /usr/local/bin/traffic_server - STACK TRACE:
          /usr/local/lib/libtsutil.so.3(ink_freelist_new+0x992)[0x2ba8ab4b5072]
          /usr/local/bin/traffic_server[0x64a68f]
          /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2)[0x64ef52]
          /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754)[0x67ef54]
          /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130)[0x6540f0]
          /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74)[0x521c14]
          /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd)[0x538c1d]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20)[0x54ffd0]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92)[0x550042]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a)[0x55091a]
          /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d)[0x4bec1d]
          /usr/local/libexec/trafficserver/purge.so(+0x1e56)[0x2ba8b99bfe56]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb)[0x54927b]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0x39f)[0x54793f]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x25a)[0x547efa]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x688)[0x54a4e8]
          /usr/local/bin/traffic_server(_ZN17HttpClientSession16state_keep_aliveEiPv+0xa8)[0x523738]
          /usr/local/bin/traffic_server[0x6b6f4a]
          /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe)[0x6aff0e]
          /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e)[0x6dfd4e]
          /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0)[0x6e06c0]
          /usr/local/bin/traffic_server[0x6df9b2]
          /lib64/libpthread.so.0(+0x6a3f)[0x2ba8ab6dca3f]
          /lib64/libc.so.6(clone+0x6d)[0x2ba8ad91a67d]
          FATAL: Failed to mmap 50331648 bytes, Cannot allocate memory <----------------------------------------------------------------- here you want?
          /usr/local/bin/traffic_server - STACK TRACE:
          /usr/local/lib/libtsutil.so.3(ink_fatal+0x88)[0x2ba8ab4b0c68]
          /usr/local/lib/libtsutil.so.3(+0x17aca)[0x2ba8ab4b3aca]
          /usr/local/lib/libtsutil.so.3(ink_freelist_new+0x992)[0x2ba8ab4b5072]
          /usr/local/bin/traffic_server[0x64a68f]
          /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2)[0x64ef52]

          Show
          jaekyung oh added a comment - i've tried again in accordance with your guide. [2ba8ae8cc700:43] [ink_queue.cc:00601] [-] all:4011MB t:1024 f:8 m:40 avg:40.0 malloc:1016 csize:32 tsize:32768 cbsize:1052672 [2ba8adbb7e20:40] [ink_queue.cc:00595] [M] all:4029MB t:767 f:87 m:86 avg:84.8 malloc:680 csize:64 tsize:4096 cbsize:266240 [2ba8adbb7e20:40] [ink_queue.cc:00601] [-] all:4029MB t:703 f:23 m:86 avg:84.8 malloc:680 csize:64 tsize:4096 cbsize:266240 [2ba8ae8cc700:03] [ink_queue.cc:00668] [F] all:4044MB t:628 f:108 m:108 avg:107.1 malloc:519 csize:70 tsize:232 cbsize:16384 [2ba8ae8cc700:03] [ink_queue.cc:00674] [-] all:4044MB t:557 f:38 m:108 avg:107.1 malloc:519 csize:70 tsize:232 cbsize:16384 /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.3(ink_freelist_new+0x992) [0x2ba8ab4b5072] /usr/local/bin/traffic_server [0x64a68f] /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2) [0x64ef52] /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754) [0x67ef54] /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130) [0x6540f0] /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74) [0x521c14] /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd) [0x538c1d] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20) [0x54ffd0] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92) [0x550042] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a) [0x55091a] /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d) [0x4bec1d] /usr/local/libexec/trafficserver/purge.so(+0x1e56) [0x2ba8b99bfe56] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb) [0x54927b] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0x39f) [0x54793f] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x25a) [0x547efa] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x688) [0x54a4e8] /usr/local/bin/traffic_server(_ZN17HttpClientSession16state_keep_aliveEiPv+0xa8) [0x523738] /usr/local/bin/traffic_server [0x6b6f4a] /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe) [0x6aff0e] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e) [0x6dfd4e] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0) [0x6e06c0] /usr/local/bin/traffic_server [0x6df9b2] /lib64/libpthread.so.0(+0x6a3f) [0x2ba8ab6dca3f] /lib64/libc.so.6(clone+0x6d) [0x2ba8ad91a67d] FATAL: Failed to mmap 50331648 bytes, Cannot allocate memory <----------------------------------------------------------------- here you want? /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.3(ink_fatal+0x88) [0x2ba8ab4b0c68] /usr/local/lib/libtsutil.so.3(+0x17aca) [0x2ba8ab4b3aca] /usr/local/lib/libtsutil.so.3(ink_freelist_new+0x992) [0x2ba8ab4b5072] /usr/local/bin/traffic_server [0x64a68f] /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2) [0x64ef52]
          Hide
          Yunkai Zhang added a comment -

          jaekyung oh

          Can you modify this funtion mmap_align() from:

          mmap_align(size_t size, size_t alignment) {
            ....
            if (result == MAP_FAILED) {
                xdump();
                ink_fatal(1, "Failed to mmap %zu bytes, %s", size, strerror(errno));
            }
            ...
          }
          

          ==>

          mmap_align(size_t size, size_t alignment) {
            ....
            if (result == MAP_FAILED) {
                xdump();
                ink_fatal(1, "Failed to mmap %zu bytes, %s", size + extra, strerror(errno));
            }
            ...
          }
          

          I want to see the real size:(size + extra) that mmap() try to allocate.

          Show
          Yunkai Zhang added a comment - jaekyung oh Can you modify this funtion mmap_align() from: mmap_align(size_t size, size_t alignment) { .... if (result == MAP_FAILED) { xdump(); ink_fatal(1, "Failed to mmap %zu bytes, %s" , size, strerror(errno)); } ... } ==> mmap_align(size_t size, size_t alignment) { .... if (result == MAP_FAILED) { xdump(); ink_fatal(1, "Failed to mmap %zu bytes, %s" , size + extra, strerror(errno)); } ... } I want to see the real size:(size + extra) that mmap() try to allocate.
          Hide
          jaekyung oh added a comment -

          and we tried again with "CONFIG proxy.config.cache.ram_cache.size INT 4000000000"

          The error shows : FATAL: Failed to mmap 8392704 bytes, Cannot allocate memory

          Show
          jaekyung oh added a comment - and we tried again with "CONFIG proxy.config.cache.ram_cache.size INT 4000000000" The error shows : FATAL: Failed to mmap 8392704 bytes, Cannot allocate memory
          Hide
          Yunkai Zhang added a comment -

          jaekyung oh

          Your PAGE_SIZE is 4K, equal to us's.

          That it to say, it's difficult to mmap 48M at that time, because there are so many memory fragment.

          Let me see,

          Maybe the alignment is too large (I use alignment to help us calculate the start address of a chunk).

          I'll try to optimize this alignment strategy, Thanks for your feedback.

          Show
          Yunkai Zhang added a comment - jaekyung oh Your PAGE_SIZE is 4K, equal to us's. That it to say, it's difficult to mmap 48M at that time, because there are so many memory fragment. Let me see, Maybe the alignment is too large (I use alignment to help us calculate the start address of a chunk). I'll try to optimize this alignment strategy, Thanks for your feedback.
          Hide
          jaekyung oh added a comment -

          in OpenSuse "getconf -a | grep PAGE" shows

          PAGESIZE 4096
          PAGE_SIZE 4096
          _AVPHYS_PAGES 1658018
          _PHYS_PAGES 4114512

          Show
          jaekyung oh added a comment - in OpenSuse "getconf -a | grep PAGE" shows PAGESIZE 4096 PAGE_SIZE 4096 _AVPHYS_PAGES 1658018 _PHYS_PAGES 4114512
          Hide
          Yunkai Zhang added a comment - - edited

          jaekyung oh

          By reading your log:

          [2b59f1d13700:40][ink_queue.cc:00674][-] all:4015MB t:566 f:22 m:85 avg:84.6 malloc:544 csize:64 tsize:4096 cbsize:266240
          >> The total memory used by InkFreeList allocator is 4015MB (including the memory used by ram_cache).

          FATAL: Failed to mmap size=16781312 bytes, Cannot allocate memory
          >> The patch try to mmap an aligned memory (align 16781312 to 2^n * PAGE_SIZE).
          >> Suppose your OS's PAGE_SIZE is 4K, the aligned size should be: 2^13 * 4K = 2^25 = 32MB, so the patch will try to mmap(size + 32MB - PAGE_SIZE) firstly.

          It seems that your OS can't malloc a continuous blocks about 48MB at that time?

          What is the PAGE_SIZE on your OS?

          Show
          Yunkai Zhang added a comment - - edited jaekyung oh By reading your log: [2b59f1d13700:40] [ink_queue.cc:00674] [-] all:4015MB t:566 f:22 m:85 avg:84.6 malloc:544 csize:64 tsize:4096 cbsize:266240 >> The total memory used by InkFreeList allocator is 4015MB (including the memory used by ram_cache). FATAL: Failed to mmap size=16781312 bytes, Cannot allocate memory >> The patch try to mmap an aligned memory (align 16781312 to 2^n * PAGE_SIZE). >> Suppose your OS's PAGE_SIZE is 4K, the aligned size should be: 2^13 * 4K = 2^25 = 32MB, so the patch will try to mmap(size + 32MB - PAGE_SIZE) firstly. It seems that your OS can't malloc a continuous blocks about 48MB at that time? What is the PAGE_SIZE on your OS?
          Hide
          jaekyung oh added a comment - - edited

          The memory was used only 48% when traffic server restart by the error.

          I tried again in accordance with your advise. but same error happened.

          [2b59f1d13700:03][ink_queue.cc:00601][-] all:3871MB t:542 f:18 m:87 avg:85.9 malloc:524 csize:70 tsize:232 cbsize:16384
          [2b59f1d13700:03][ink_queue.cc:00595][M] all:3908MB t:612 f:89 m:88 avg:88.0 malloc:523 csize:70 tsize:232 cbsize:16384
          [2b59f1d13700:03][ink_queue.cc:00601][-] all:3908MB t:542 f:19 m:88 avg:88.0 malloc:523 csize:70 tsize:232 cbsize:16384
          [2b59f1c12700:03][ink_queue.cc:00595][M] all:3912MB t:617 f:90 m:89 avg:88.8 malloc:527 csize:70 tsize:232 cbsize:16384
          [2b59f1c12700:03][ink_queue.cc:00601][-] all:3912MB t:547 f:20 m:89 avg:88.8 malloc:527 csize:70 tsize:232 cbsize:16384
          [2b59fd7b4700:02][ink_queue.cc:00595][M] all:3918MB t:3640 f:224 m:224 avg:212.0 malloc:3416 csize:170 tsize:72 cbsize:12288
          [2b59fd7b4700:02][ink_queue.cc:00601][-] all:3918MB t:3470 f:54 m:224 avg:212.0 malloc:3416 csize:170 tsize:72 cbsize:12288
          [2b59f1202e20:01][ink_queue.cc:00595][M] all:3918MB t:9765 f:384 m:384 avg:381.4 malloc:9381 csize:279 tsize:88 cbsize:24576
          [2b59f1202e20:01][ink_queue.cc:00601][-] all:3918MB t:9486 f:105 m:384 avg:381.4 malloc:9381 csize:279 tsize:88 cbsize:24576
          [2b59f1e14700:01][ink_queue.cc:00668][F] all:3983MB t:10043 f:404 m:404 avg:363.3 malloc:9638 csize:279 tsize:88 cbsize:24576
          [2b59f1e14700:01][ink_queue.cc:00674][-] all:3983MB t:9763 f:125 m:404 avg:363.3 malloc:9638 csize:279 tsize:88 cbsize:24576
          [2b59f1d13700:01][ink_queue.cc:00595][M] all:3983MB t:10075 f:352 m:352 avg:324.7 malloc:9723 csize:279 tsize:88 cbsize:24576
          [2b59f1d13700:01][ink_queue.cc:00601][-] all:3983MB t:9796 f:73 m:352 avg:324.7 malloc:9723 csize:279 tsize:88 cbsize:24576
          [2b59f1c12700:43][ink_queue.cc:00595][M] all:3983MB t:1180 f:40 m:38 avg:38.0 malloc:1140 csize:32 tsize:32768 cbsize:1052672
          [2b59f1c12700:43][ink_queue.cc:00601][-] all:3983MB t:1148 f:8 m:38 avg:38.0 malloc:1140 csize:32 tsize:32768 cbsize:1052672
          [2b59f1d13700:43][ink_queue.cc:00595][M] all:3983MB t:1272 f:39 m:37 avg:37.0 malloc:1233 csize:32 tsize:32768 cbsize:1052672
          [2b59f1d13700:43][ink_queue.cc:00601][-] all:3983MB t:1240 f:7 m:37 avg:37.0 malloc:1233 csize:32 tsize:32768 cbsize:1052672
          [2b59f1c12700:03][ink_queue.cc:00595][M] all:3984MB t:617 f:91 m:89 avg:87.7 malloc:526 csize:70 tsize:232 cbsize:16384
          [2b59f1c12700:03][ink_queue.cc:00601][-] all:3984MB t:547 f:21 m:89 avg:87.7 malloc:526 csize:70 tsize:232 cbsize:16384
          [2b59f1c12700:03][ink_queue.cc:00668][F] all:4010MB t:617 f:81 m:81 avg:81.0 malloc:535 csize:70 tsize:232 cbsize:16384
          [2b59f1c12700:03][ink_queue.cc:00674][-] all:4010MB t:546 f:11 m:81 avg:81.0 malloc:535 csize:70 tsize:232 cbsize:16384
          [2b59f1202e20:03][ink_queue.cc:00595][M] all:4010MB t:611 f:84 m:83 avg:83.0 malloc:527 csize:70 tsize:232 cbsize:16384
          [2b59f1202e20:03][ink_queue.cc:00601][-] all:4010MB t:541 f:14 m:83 avg:83.0 malloc:527 csize:70 tsize:232 cbsize:16384
          [2b59f1e14700:04][ink_queue.cc:00595][M] all:4010MB t:9773 f:107 m:106 avg:104.6 malloc:9666 csize:85 tsize:48 cbsize:4096
          [2b59f1e14700:04][ink_queue.cc:00601][-] all:4010MB t:9688 f:22 m:106 avg:104.6 malloc:9666 csize:85 tsize:48 cbsize:4096
          [2b59f1d13700:33][ink_queue.cc:00595][M] all:4010MB t:668 f:161 m:151 avg:149.9 malloc:507 csize:134 tsize:640 cbsize:86016
          [2b59f1d13700:33][ink_queue.cc:00601][-] all:4010MB t:534 f:27 m:151 avg:149.9 malloc:507 csize:134 tsize:640 cbsize:86016
          [2b59f2016700:43][ink_queue.cc:00595][M] all:4010MB t:1274 f:52 m:50 avg:50.0 malloc:1222 csize:32 tsize:32768 cbsize:1052672
          [2b59f2016700:43][ink_queue.cc:00601][-] all:4010MB t:1242 f:20 m:50 avg:50.0 malloc:1222 csize:32 tsize:32768 cbsize:1052672
          [2b59f1c12700:43][ink_queue.cc:00668][F] all:4010MB t:1180 f:33 m:33 avg:33.0 malloc:1146 csize:32 tsize:32768 cbsize:1052672
          [2b59f1c12700:43][ink_queue.cc:00674][-] all:4010MB t:1147 f:1 m:33 avg:33.0 malloc:1146 csize:32 tsize:32768 cbsize:1052672
          [2b59f1202e20:04][ink_queue.cc:00668][F] all:4015MB t:9519 f:114 m:114 avg:113.2 malloc:9404 csize:85 tsize:48 cbsize:4096
          [2b59f1202e20:04][ink_queue.cc:00674][-] all:4015MB t:9433 f:29 m:114 avg:113.2 malloc:9404 csize:85 tsize:48 cbsize:4096
          [2b59f1d13700:40][ink_queue.cc:00668][F] all:4015MB t:631 f:86 m:85 avg:84.6 malloc:544 csize:64 tsize:4096 cbsize:266240
          [2b59f1d13700:40][ink_queue.cc:00674][-] all:4015MB t:566 f:22 m:85 avg:84.6 malloc:544 csize:64 tsize:4096 cbsize:266240
          /usr/local/bin/traffic_server - STACK TRACE:
          /usr/local/bin/traffic_server[0x64a68f]
          /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2)[0x64ef52]
          /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754)[0x67ef54]
          /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130)[0x6540f0]
          /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74)[0x521c14]
          /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd)[0x538c1d]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20)[0x54ffd0]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92)[0x550042]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a)[0x55091a]
          /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d)[0x4bec1d]
          /usr/local/libexec/trafficserver/purge.so(+0x1e56)[0x2b59fd9bfe56]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb)[0x54927b]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0x39f)[0x54793f]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x25a)[0x547efa]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x688)[0x54a4e8]
          /usr/local/bin/traffic_server(_ZN17HttpClientSession16state_keep_aliveEiPv+0xa8)[0x523738]
          /usr/local/bin/traffic_server[0x6b6f4a]
          /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe)[0x6aff0e]
          /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e)[0x6dfd4e]
          /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0)[0x6e06c0]
          /usr/local/bin/traffic_server[0x6df9b2]
          /lib64/libpthread.so.0(+0x6a3f)[0x2b59eed27a3f]
          /lib64/libc.so.6(clone+0x6d)[0x2b59f0f6567d]
          FATAL: Failed to mmap 16781312 bytes, Cannot allocate memory
          /usr/local/bin/traffic_server - STACK TRACE:
          /usr/local/lib/libtsutil.so.3(ink_fatal+0x88)[0x2b59eeafbc68]
          /usr/local/lib/libtsutil.so.3(ink_freelist_new+0xc09)[0x2b59eeb001f9]
          /usr/local/bin/traffic_server[0x64a68f]
          /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2)[0x64ef52]
          /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754)[0x67ef54]
          /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130)[0x6540f0]
          /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74)[0x521c14]
          /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd)[0x538c1d]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20)[0x54ffd0]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92)[0x550042]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a)[0x55091a]
          /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d)[0x4bec1d]
          /usr/local/libexec/trafficserver/purge.so(+0x1e56)[0x2b59fd9bfe56]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb)[0x54927b]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0x39f)[0x54793f]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x25a)[0x547efa]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x688)[0x54a4e8]
          /usr/local/bin/traffic_server(_ZN17HttpClientSession16state_keep_aliveEiPv+0xa8)[0x523738]
          /usr/local/bin/traffic_server[0x6b6f4a]
          /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe)[0x6aff0e]
          /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e)[0x6dfd4e]
          /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0)[0x6e06c0]
          /usr/local/bin/traffic_server[0x6df9b2]
          /lib64/libpthread.so.0(+0x6a3f)[0x2b59eed27a3f]
          /lib64/libc.so.6(clone+0x6d)[0x2b59f0f6567d]
          [Dec 24 15:23:46.343] Manager

          {0x7fe0adc5c720} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
          [Dec 24 15:23:46.343] Manager {0x7fe0adc5c720}

          FATAL: (last system error 104: Connection reset by peer)
          [Dec 24 15:23:46.343] Manager

          {0x7fe0adc5c720} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
          [Dec 24 15:23:46.343] Manager {0x7fe0adc5c720}

          NOTE: [LocalManager::processShutdown] Executing process shutdown request.

          maybe applying your patches onto ATS 3.2.0 seems to not working..

          and the error happens around 4G byte memory....does it mean something?

          Show
          jaekyung oh added a comment - - edited The memory was used only 48% when traffic server restart by the error. I tried again in accordance with your advise. but same error happened. [2b59f1d13700:03] [ink_queue.cc:00601] [-] all:3871MB t:542 f:18 m:87 avg:85.9 malloc:524 csize:70 tsize:232 cbsize:16384 [2b59f1d13700:03] [ink_queue.cc:00595] [M] all:3908MB t:612 f:89 m:88 avg:88.0 malloc:523 csize:70 tsize:232 cbsize:16384 [2b59f1d13700:03] [ink_queue.cc:00601] [-] all:3908MB t:542 f:19 m:88 avg:88.0 malloc:523 csize:70 tsize:232 cbsize:16384 [2b59f1c12700:03] [ink_queue.cc:00595] [M] all:3912MB t:617 f:90 m:89 avg:88.8 malloc:527 csize:70 tsize:232 cbsize:16384 [2b59f1c12700:03] [ink_queue.cc:00601] [-] all:3912MB t:547 f:20 m:89 avg:88.8 malloc:527 csize:70 tsize:232 cbsize:16384 [2b59fd7b4700:02] [ink_queue.cc:00595] [M] all:3918MB t:3640 f:224 m:224 avg:212.0 malloc:3416 csize:170 tsize:72 cbsize:12288 [2b59fd7b4700:02] [ink_queue.cc:00601] [-] all:3918MB t:3470 f:54 m:224 avg:212.0 malloc:3416 csize:170 tsize:72 cbsize:12288 [2b59f1202e20:01] [ink_queue.cc:00595] [M] all:3918MB t:9765 f:384 m:384 avg:381.4 malloc:9381 csize:279 tsize:88 cbsize:24576 [2b59f1202e20:01] [ink_queue.cc:00601] [-] all:3918MB t:9486 f:105 m:384 avg:381.4 malloc:9381 csize:279 tsize:88 cbsize:24576 [2b59f1e14700:01] [ink_queue.cc:00668] [F] all:3983MB t:10043 f:404 m:404 avg:363.3 malloc:9638 csize:279 tsize:88 cbsize:24576 [2b59f1e14700:01] [ink_queue.cc:00674] [-] all:3983MB t:9763 f:125 m:404 avg:363.3 malloc:9638 csize:279 tsize:88 cbsize:24576 [2b59f1d13700:01] [ink_queue.cc:00595] [M] all:3983MB t:10075 f:352 m:352 avg:324.7 malloc:9723 csize:279 tsize:88 cbsize:24576 [2b59f1d13700:01] [ink_queue.cc:00601] [-] all:3983MB t:9796 f:73 m:352 avg:324.7 malloc:9723 csize:279 tsize:88 cbsize:24576 [2b59f1c12700:43] [ink_queue.cc:00595] [M] all:3983MB t:1180 f:40 m:38 avg:38.0 malloc:1140 csize:32 tsize:32768 cbsize:1052672 [2b59f1c12700:43] [ink_queue.cc:00601] [-] all:3983MB t:1148 f:8 m:38 avg:38.0 malloc:1140 csize:32 tsize:32768 cbsize:1052672 [2b59f1d13700:43] [ink_queue.cc:00595] [M] all:3983MB t:1272 f:39 m:37 avg:37.0 malloc:1233 csize:32 tsize:32768 cbsize:1052672 [2b59f1d13700:43] [ink_queue.cc:00601] [-] all:3983MB t:1240 f:7 m:37 avg:37.0 malloc:1233 csize:32 tsize:32768 cbsize:1052672 [2b59f1c12700:03] [ink_queue.cc:00595] [M] all:3984MB t:617 f:91 m:89 avg:87.7 malloc:526 csize:70 tsize:232 cbsize:16384 [2b59f1c12700:03] [ink_queue.cc:00601] [-] all:3984MB t:547 f:21 m:89 avg:87.7 malloc:526 csize:70 tsize:232 cbsize:16384 [2b59f1c12700:03] [ink_queue.cc:00668] [F] all:4010MB t:617 f:81 m:81 avg:81.0 malloc:535 csize:70 tsize:232 cbsize:16384 [2b59f1c12700:03] [ink_queue.cc:00674] [-] all:4010MB t:546 f:11 m:81 avg:81.0 malloc:535 csize:70 tsize:232 cbsize:16384 [2b59f1202e20:03] [ink_queue.cc:00595] [M] all:4010MB t:611 f:84 m:83 avg:83.0 malloc:527 csize:70 tsize:232 cbsize:16384 [2b59f1202e20:03] [ink_queue.cc:00601] [-] all:4010MB t:541 f:14 m:83 avg:83.0 malloc:527 csize:70 tsize:232 cbsize:16384 [2b59f1e14700:04] [ink_queue.cc:00595] [M] all:4010MB t:9773 f:107 m:106 avg:104.6 malloc:9666 csize:85 tsize:48 cbsize:4096 [2b59f1e14700:04] [ink_queue.cc:00601] [-] all:4010MB t:9688 f:22 m:106 avg:104.6 malloc:9666 csize:85 tsize:48 cbsize:4096 [2b59f1d13700:33] [ink_queue.cc:00595] [M] all:4010MB t:668 f:161 m:151 avg:149.9 malloc:507 csize:134 tsize:640 cbsize:86016 [2b59f1d13700:33] [ink_queue.cc:00601] [-] all:4010MB t:534 f:27 m:151 avg:149.9 malloc:507 csize:134 tsize:640 cbsize:86016 [2b59f2016700:43] [ink_queue.cc:00595] [M] all:4010MB t:1274 f:52 m:50 avg:50.0 malloc:1222 csize:32 tsize:32768 cbsize:1052672 [2b59f2016700:43] [ink_queue.cc:00601] [-] all:4010MB t:1242 f:20 m:50 avg:50.0 malloc:1222 csize:32 tsize:32768 cbsize:1052672 [2b59f1c12700:43] [ink_queue.cc:00668] [F] all:4010MB t:1180 f:33 m:33 avg:33.0 malloc:1146 csize:32 tsize:32768 cbsize:1052672 [2b59f1c12700:43] [ink_queue.cc:00674] [-] all:4010MB t:1147 f:1 m:33 avg:33.0 malloc:1146 csize:32 tsize:32768 cbsize:1052672 [2b59f1202e20:04] [ink_queue.cc:00668] [F] all:4015MB t:9519 f:114 m:114 avg:113.2 malloc:9404 csize:85 tsize:48 cbsize:4096 [2b59f1202e20:04] [ink_queue.cc:00674] [-] all:4015MB t:9433 f:29 m:114 avg:113.2 malloc:9404 csize:85 tsize:48 cbsize:4096 [2b59f1d13700:40] [ink_queue.cc:00668] [F] all:4015MB t:631 f:86 m:85 avg:84.6 malloc:544 csize:64 tsize:4096 cbsize:266240 [2b59f1d13700:40] [ink_queue.cc:00674] [-] all:4015MB t:566 f:22 m:85 avg:84.6 malloc:544 csize:64 tsize:4096 cbsize:266240 /usr/local/bin/traffic_server - STACK TRACE: /usr/local/bin/traffic_server [0x64a68f] /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2) [0x64ef52] /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754) [0x67ef54] /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130) [0x6540f0] /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74) [0x521c14] /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd) [0x538c1d] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20) [0x54ffd0] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92) [0x550042] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a) [0x55091a] /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d) [0x4bec1d] /usr/local/libexec/trafficserver/purge.so(+0x1e56) [0x2b59fd9bfe56] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb) [0x54927b] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0x39f) [0x54793f] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x25a) [0x547efa] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x688) [0x54a4e8] /usr/local/bin/traffic_server(_ZN17HttpClientSession16state_keep_aliveEiPv+0xa8) [0x523738] /usr/local/bin/traffic_server [0x6b6f4a] /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe) [0x6aff0e] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e) [0x6dfd4e] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0) [0x6e06c0] /usr/local/bin/traffic_server [0x6df9b2] /lib64/libpthread.so.0(+0x6a3f) [0x2b59eed27a3f] /lib64/libc.so.6(clone+0x6d) [0x2b59f0f6567d] FATAL: Failed to mmap 16781312 bytes, Cannot allocate memory /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.3(ink_fatal+0x88) [0x2b59eeafbc68] /usr/local/lib/libtsutil.so.3(ink_freelist_new+0xc09) [0x2b59eeb001f9] /usr/local/bin/traffic_server [0x64a68f] /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x1e2) [0x64ef52] /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754) [0x67ef54] /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130) [0x6540f0] /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74) [0x521c14] /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd) [0x538c1d] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20) [0x54ffd0] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92) [0x550042] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a) [0x55091a] /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d) [0x4bec1d] /usr/local/libexec/trafficserver/purge.so(+0x1e56) [0x2b59fd9bfe56] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb) [0x54927b] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0x39f) [0x54793f] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x25a) [0x547efa] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x688) [0x54a4e8] /usr/local/bin/traffic_server(_ZN17HttpClientSession16state_keep_aliveEiPv+0xa8) [0x523738] /usr/local/bin/traffic_server [0x6b6f4a] /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe) [0x6aff0e] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e) [0x6dfd4e] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0) [0x6e06c0] /usr/local/bin/traffic_server [0x6df9b2] /lib64/libpthread.so.0(+0x6a3f) [0x2b59eed27a3f] /lib64/libc.so.6(clone+0x6d) [0x2b59f0f6567d] [Dec 24 15:23:46.343] Manager {0x7fe0adc5c720} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) [Dec 24 15:23:46.343] Manager {0x7fe0adc5c720} FATAL: (last system error 104: Connection reset by peer) [Dec 24 15:23:46.343] Manager {0x7fe0adc5c720} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request. [Dec 24 15:23:46.343] Manager {0x7fe0adc5c720} NOTE: [LocalManager::processShutdown] Executing process shutdown request. maybe applying your patches onto ATS 3.2.0 seems to not working.. and the error happens around 4G byte memory....does it mean something?
          Hide
          Yunkai Zhang added a comment - - edited

          jaekyung oh

          It seems that mmap() failed to allocate memory. Was OS memory used up at that time?

          This patches was based on master branch, you should also apply it carefully.

          I haven't test these on OpenSuse.

          And, you can enable debug info by modify debug_filter's value(do you notice it?):

          ##############################################################################
          #
          # Configuration for InkFreeList memory allocator
          #
          ##############################################################################
            # Dump debug information according bit mask of debug_filter, if a bit is set
            # in the mask, then debug information of the corresponding action are dumped:
            #  bit 0: reclaim memory in ink_freelist_new
            #  bit 1: reclaim memory in ink_freelist_free
            #  bit 2: fetch memory from thread cache
          CONFIG proxy.config.allocator.debug_filter INT 3
            # The value of enable_reclaim should be 0 or 1. We can disable all reclaiming
            # strategy by setting it to be 0.
          CONFIG proxy.config.allocator.enable_reclaim INT 1
            # The value of reclaim_factor should be in [0, 1], allocator use it to
            # calculate average value of idle memory in InkFreeList, which will determine
            # when to reclaim memory. The larger the value, the faster the reclaiming.
            # This value is effective only when enable_reclaim is 1.
          CONFIG proxy.config.allocator.reclaim_factor FLOAT 0.200000
            # Allocator will reclaim memory only when it continuously satisfy the reclaim
            # condition for max_overage times. This value is effective only when
            # enable_reclaim is 1.
          CONFIG proxy.config.allocator.max_overage INT 100
          
          Show
          Yunkai Zhang added a comment - - edited jaekyung oh It seems that mmap() failed to allocate memory. Was OS memory used up at that time? This patches was based on master branch, you should also apply it carefully. I haven't test these on OpenSuse. And, you can enable debug info by modify debug_filter's value(do you notice it?): ############################################################################## # # Configuration for InkFreeList memory allocator # ############################################################################## # Dump debug information according bit mask of debug_filter, if a bit is set # in the mask, then debug information of the corresponding action are dumped: # bit 0: reclaim memory in ink_freelist_new # bit 1: reclaim memory in ink_freelist_free # bit 2: fetch memory from thread cache CONFIG proxy.config.allocator.debug_filter INT 3 # The value of enable_reclaim should be 0 or 1. We can disable all reclaiming # strategy by setting it to be 0. CONFIG proxy.config.allocator.enable_reclaim INT 1 # The value of reclaim_factor should be in [0, 1], allocator use it to # calculate average value of idle memory in InkFreeList, which will determine # when to reclaim memory. The larger the value, the faster the reclaiming. # This value is effective only when enable_reclaim is 1. CONFIG proxy.config.allocator.reclaim_factor FLOAT 0.200000 # Allocator will reclaim memory only when it continuously satisfy the reclaim # condition for max_overage times. This value is effective only when # enable_reclaim is 1. CONFIG proxy.config.allocator.max_overage INT 100
          Hide
          jaekyung oh added a comment -

          i applied the patches onto ats 3.2.0 and i've got these errors.

          OS : OpenSuse 11.4
          RAM : 16G
          config :
          CONFIG proxy.config.cache.ram_cache.size INT 8000000000
          CONFIG proxy.config.cache.ram_cache_cutoff INT 6000000000

          traffic.out
          [Dec 21 19:05:00.070] Server

          {0x2ba0c51d1700}

          STATUS: The logfile /usr/local/var/log/trafficserver/squid.log was rolled
          to /usr/local/var/log/trafficserver/squid.log_PM0010-Z001-R001.20121221.18h59m55s-20121221.19h05m00s.old.
          /usr/local/bin/traffic_server - STACK TRACE:
          /usr/local/bin/traffic_server[0x64a68f]
          /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x51a)[0x64f28a]
          /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754)[0x67ef54]
          /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130)[0x6540f0]
          /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74)[0x521c14]
          /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd)[0x538c1d]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20)[0x54ffd0]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92)[0x550042]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a)[0x55091a]
          /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d)[0x4bec1d]
          /usr/local/libexec/trafficserver/purge.so(+0x1e56)[0x2ba0c5a3fe56]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb)[0x54927b]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x84f)[0x54723f]
          /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8)[0x54d208]
          /usr/local/bin/traffic_server[0x6b6f4a]
          /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe)[0x6aff0e]
          /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e)[0x6dfd4e]
          /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0)[0x6e06c0]
          /usr/local/bin/traffic_server(main+0xdee)[0x4cc2ce]
          /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ba0b976cbfd]
          /usr/local/bin/traffic_server[0x48a369]
          FATAL: Failed to mmap 4198400 bytes, Cannot allocate memory
          /usr/local/bin/traffic_server - STACK TRACE:
          /usr/local/lib/libtsutil.so.3(ink_fatal+0x88)[0x2ba0b73b7c68]
          /usr/local/lib/libtsutil.so.3(ink_freelist_new+0xc09)[0x2ba0b73bc1f9]
          /usr/local/bin/traffic_server[0x64a68f]
          /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x51a)[0x64f28a]
          /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754)[0x67ef54]
          /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130)[0x6540f0]
          /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74)[0x521c14]
          /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd)[0x538c1d]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20)[0x54ffd0]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92)[0x550042]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a)[0x547fda]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353)[0x5494d3]
          /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a)[0x55091a]
          /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d)[0x4bec1d]
          /usr/local/libexec/trafficserver/purge.so(+0x1e56)[0x2ba0c5a3fe56]
          /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb)[0x54927b]
          /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538)[0x54f9e8]
          /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x84f)[0x54723f]
          /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8)[0x54d208]
          /usr/local/bin/traffic_server[0x6b6f4a]
          /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe)[0x6aff0e]
          /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e)[0x6dfd4e]
          /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0)[0x6e06c0]
          /usr/local/bin/traffic_server(main+0xdee)[0x4cc2ce]
          /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ba0b976cbfd]
          /usr/local/bin/traffic_server[0x48a369]
          [Dec 21 19:05:29.767] Manager

          {0x7f730822e720} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
          [Dec 21 19:05:29.767] Manager {0x7f730822e720}

          FATAL: (last system error 104: Connection reset by peer)
          [Dec 21 19:05:29.767] Manager

          {0x7f730822e720} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
          [Dec 21 19:05:29.767] Manager {0x7f730822e720}

          NOTE: [LocalManager::processShutdown] Executing process shutdown
          request.
          [Dec 21 19:05:29.767] Manager

          {0x7f730822e720} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
          [Dec 21 19:05:29.767] Manager {0x7f730822e720}

          ERROR: (last system error 32: Broken pipe)
          [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
          [Dec 21 19:05:29.799]

          {0x7f3b1cf41720} STATUS: opened /usr/local/var/log/trafficserver/manager.log
          [Dec 21 19:05:29.799] {0x7f3b1cf41720}

          NOTE: updated diags config
          [Dec 21 19:05:29.799] Manager

          {0x7f3b1cf41720} NOTE: [appendDefaultDomain] Unable to determine default domain name.
          Nodes will be know by their unqualified host name
          [Dec 21 19:05:29.800] Manager {0x7f3b1cf41720}

          NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release:
          '2.6.37.1-1.2-desktop'
          [Dec 21 19:05:29.801] Manager

          {0x7f3b1cf41720} NOTE: [LocalManager::listenForProxy] Listening on port: 80
          [Dec 21 19:05:29.801] Manager {0x7f3b1cf41720}

          NOTE: [LocalManager::listenForProxy] Listening on port: 443
          [Dec 21 19:05:29.801] Manager

          {0x7f3b1cf41720} NOTE: [TrafficManager] Setup complete
          [Dec 21 19:05:30.816] Manager {0x7f3b1cf41720}

          NOTE: [LocalManager::startProxy] Launching ts process
          [TrafficServer] using root directory '/usr/local'
          [Dec 21 19:05:30.825] Manager

          {0x7f3b1cf41720} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd
          '14'
          [Dec 21 19:05:30.825] Manager {0x7f3b1cf41720}

          NOTE: [Alarms::signalAlarm] Server Process born
          [Dec 21 19:05:31.843]

          {0x2b9e0643ce20} STATUS: opened /usr/local/var/log/trafficserver/diags.log
          [Dec 21 19:05:31.843] {0x2b9e0643ce20}

          NOTE: updated diags config
          [Dec 21 19:05:31.848] Server

          {0x2b9e0643ce20} NOTE: cache clustering disabled
          [Dec 21 19:05:31.861] Server {0x2b9e0643ce20}

          NOTE: cache clustering disabled
          [Dec 21 19:05:31.881] Server

          {0x2b9e0643ce20} NOTE: logging initialized[15], logging_mode = 3
          [Dec 21 19:05:31.890] Server {0x2b9e0643ce20}

          NOTE: loading plugin '/usr/local/libexec/trafficserver/purge.so'
          [Dec 21 19:05:31.891] Server

          {0x2b9e0643ce20} NOTE: loading plugin '/usr/local/libexec/trafficserver/remap_bgcache.so'
          [Dec 21 19:05:31.893] Server {0x2b9e0643ce20}

          NOTE: traffic server running
          NOTE: Traffic Server received Sig 11: Segmentation fault
          /usr/local/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0xf2d0)[0x2b9e03f6a2d0]
          /usr/local/bin/traffic_server(_Z15dir_clear_rangellP3Vol+0x6a)[0x65ac5a]
          /usr/local/bin/traffic_server(_ZN3Vol24handle_recover_from_dataEiPv+0x4de)[0x65188e]
          /usr/local/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x35)[0x656415]
          /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e)[0x6dfd4e]
          /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x5ab)[0x6e077b]
          /usr/local/bin/traffic_server[0x6df9b2]
          /lib64/libpthread.so.0(+0x6a3f)[0x2b9e03f61a3f]
          /lib64/libc.so.6(clone+0x6d)[0x2b9e0619f67d]
          [Dec 21 19:06:06.183] Manager

          {0x7f3b1cf41720} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
          [Dec 21 19:06:06.183] Manager {0x7f3b1cf41720}

          FATAL: (last system error 104: Connection reset by peer)
          [Dec 21 19:06:06.183] Manager

          {0x7f3b1cf41720} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
          [Dec 21 19:06:06.183] Manager {0x7f3b1cf41720}

          NOTE: [LocalManager::processShutdown] Executing process shutdown request.
          .
          .
          .
          . (repeat restarting by "dir_clear_rangellP3Vol" error)

          any hint? should i not apply the patches on ats 3.2.0?

          Show
          jaekyung oh added a comment - i applied the patches onto ats 3.2.0 and i've got these errors. OS : OpenSuse 11.4 RAM : 16G config : CONFIG proxy.config.cache.ram_cache.size INT 8000000000 CONFIG proxy.config.cache.ram_cache_cutoff INT 6000000000 traffic.out [Dec 21 19:05:00.070] Server {0x2ba0c51d1700} STATUS: The logfile /usr/local/var/log/trafficserver/squid.log was rolled to /usr/local/var/log/trafficserver/squid.log_PM0010-Z001-R001.20121221.18h59m55s-20121221.19h05m00s.old. /usr/local/bin/traffic_server - STACK TRACE: /usr/local/bin/traffic_server [0x64a68f] /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x51a) [0x64f28a] /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754) [0x67ef54] /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130) [0x6540f0] /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74) [0x521c14] /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd) [0x538c1d] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20) [0x54ffd0] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92) [0x550042] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a) [0x55091a] /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d) [0x4bec1d] /usr/local/libexec/trafficserver/purge.so(+0x1e56) [0x2ba0c5a3fe56] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb) [0x54927b] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x84f) [0x54723f] /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8) [0x54d208] /usr/local/bin/traffic_server [0x6b6f4a] /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe) [0x6aff0e] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e) [0x6dfd4e] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0) [0x6e06c0] /usr/local/bin/traffic_server(main+0xdee) [0x4cc2ce] /lib64/libc.so.6(__libc_start_main+0xfd) [0x2ba0b976cbfd] /usr/local/bin/traffic_server [0x48a369] FATAL: Failed to mmap 4198400 bytes, Cannot allocate memory /usr/local/bin/traffic_server - STACK TRACE: /usr/local/lib/libtsutil.so.3(ink_fatal+0x88) [0x2ba0b73b7c68] /usr/local/lib/libtsutil.so.3(ink_freelist_new+0xc09) [0x2ba0b73bc1f9] /usr/local/bin/traffic_server [0x64a68f] /usr/local/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x51a) [0x64f28a] /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x754) [0x67ef54] /usr/local/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0x130) [0x6540f0] /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x74) [0x521c14] /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0xfd) [0x538c1d] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb20) [0x54ffd0] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xb92) [0x550042] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x33a) [0x547fda] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x353) [0x5494d3] /usr/local/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x9a) [0x55091a] /usr/local/bin/traffic_server(TSHttpTxnReenable+0x40d) [0x4bec1d] /usr/local/libexec/trafficserver/purge.so(+0x1e56) [0x2ba0c5a3fe56] /usr/local/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xfb) [0x54927b] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x538) [0x54f9e8] /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x84f) [0x54723f] /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8) [0x54d208] /usr/local/bin/traffic_server [0x6b6f4a] /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1fe) [0x6aff0e] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e) [0x6dfd4e] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4f0) [0x6e06c0] /usr/local/bin/traffic_server(main+0xdee) [0x4cc2ce] /lib64/libc.so.6(__libc_start_main+0xfd) [0x2ba0b976cbfd] /usr/local/bin/traffic_server [0x48a369] [Dec 21 19:05:29.767] Manager {0x7f730822e720} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) [Dec 21 19:05:29.767] Manager {0x7f730822e720} FATAL: (last system error 104: Connection reset by peer) [Dec 21 19:05:29.767] Manager {0x7f730822e720} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request. [Dec 21 19:05:29.767] Manager {0x7f730822e720} NOTE: [LocalManager::processShutdown] Executing process shutdown request. [Dec 21 19:05:29.767] Manager {0x7f730822e720} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message [Dec 21 19:05:29.767] Manager {0x7f730822e720} ERROR: (last system error 32: Broken pipe) [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local' [Dec 21 19:05:29.799] {0x7f3b1cf41720} STATUS: opened /usr/local/var/log/trafficserver/manager.log [Dec 21 19:05:29.799] {0x7f3b1cf41720} NOTE: updated diags config [Dec 21 19:05:29.799] Manager {0x7f3b1cf41720} NOTE: [appendDefaultDomain] Unable to determine default domain name. Nodes will be know by their unqualified host name [Dec 21 19:05:29.800] Manager {0x7f3b1cf41720} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.37.1-1.2-desktop' [Dec 21 19:05:29.801] Manager {0x7f3b1cf41720} NOTE: [LocalManager::listenForProxy] Listening on port: 80 [Dec 21 19:05:29.801] Manager {0x7f3b1cf41720} NOTE: [LocalManager::listenForProxy] Listening on port: 443 [Dec 21 19:05:29.801] Manager {0x7f3b1cf41720} NOTE: [TrafficManager] Setup complete [Dec 21 19:05:30.816] Manager {0x7f3b1cf41720} NOTE: [LocalManager::startProxy] Launching ts process [TrafficServer] using root directory '/usr/local' [Dec 21 19:05:30.825] Manager {0x7f3b1cf41720} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '14' [Dec 21 19:05:30.825] Manager {0x7f3b1cf41720} NOTE: [Alarms::signalAlarm] Server Process born [Dec 21 19:05:31.843] {0x2b9e0643ce20} STATUS: opened /usr/local/var/log/trafficserver/diags.log [Dec 21 19:05:31.843] {0x2b9e0643ce20} NOTE: updated diags config [Dec 21 19:05:31.848] Server {0x2b9e0643ce20} NOTE: cache clustering disabled [Dec 21 19:05:31.861] Server {0x2b9e0643ce20} NOTE: cache clustering disabled [Dec 21 19:05:31.881] Server {0x2b9e0643ce20} NOTE: logging initialized [15] , logging_mode = 3 [Dec 21 19:05:31.890] Server {0x2b9e0643ce20} NOTE: loading plugin '/usr/local/libexec/trafficserver/purge.so' [Dec 21 19:05:31.891] Server {0x2b9e0643ce20} NOTE: loading plugin '/usr/local/libexec/trafficserver/remap_bgcache.so' [Dec 21 19:05:31.893] Server {0x2b9e0643ce20} NOTE: traffic server running NOTE: Traffic Server received Sig 11: Segmentation fault /usr/local/bin/traffic_server - STACK TRACE: /lib64/libpthread.so.0(+0xf2d0) [0x2b9e03f6a2d0] /usr/local/bin/traffic_server(_Z15dir_clear_rangellP3Vol+0x6a) [0x65ac5a] /usr/local/bin/traffic_server(_ZN3Vol24handle_recover_from_dataEiPv+0x4de) [0x65188e] /usr/local/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x35) [0x656415] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8e) [0x6dfd4e] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x5ab) [0x6e077b] /usr/local/bin/traffic_server [0x6df9b2] /lib64/libpthread.so.0(+0x6a3f) [0x2b9e03f61a3f] /lib64/libc.so.6(clone+0x6d) [0x2b9e0619f67d] [Dec 21 19:06:06.183] Manager {0x7f3b1cf41720} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) [Dec 21 19:06:06.183] Manager {0x7f3b1cf41720} FATAL: (last system error 104: Connection reset by peer) [Dec 21 19:06:06.183] Manager {0x7f3b1cf41720} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request. [Dec 21 19:06:06.183] Manager {0x7f3b1cf41720} NOTE: [LocalManager::processShutdown] Executing process shutdown request. . . . . (repeat restarting by "dir_clear_rangellP3Vol" error) any hint? should i not apply the patches on ats 3.2.0?
          Hide
          Yunkai Zhang added a comment -

          [PATCH V3 1/2] Allocator: optimize InkFreeList memory pool
          V3:

          • Remove "max_mem_in_mb" option, not limit the max memory in InkFreeList. For more thought and dicussing, It would be better to limit memory in upper layer.

          [PATCH V3 2/2] Allocator: make InkFreeList memory pool configurable

          V3:

          • Remove "max_mem_in_mb" option, not limit the max memory used by upper layer.
          • Add a new option: "enable_reclaim", which indicates enable/disable reclaiming strategy, and make the its defa
          Show
          Yunkai Zhang added a comment - [PATCH V3 1/2] Allocator: optimize InkFreeList memory pool V3: Remove "max_mem_in_mb" option, not limit the max memory in InkFreeList. For more thought and dicussing, It would be better to limit memory in upper layer. [PATCH V3 2/2] Allocator: make InkFreeList memory pool configurable V3: Remove "max_mem_in_mb" option, not limit the max memory used by upper layer. Add a new option: "enable_reclaim", which indicates enable/disable reclaiming strategy, and make the its defa
          Hide
          Yunkai Zhang added a comment -

          [PATH V2 1/2] Allocator: optimize InkFreeList memory pool
          [PATH V2 2/2] Allocator: make InkFreeList memory pool configurable

          ChangeLog:
          1) By using alignment method, we can caclutate chunk address directly, instread of store a pointer back to InkChunkInfo in each allocation.
          2) make the default "max_mem" to be 0, which means never try to reclaim memory in any time.

          Show
          Yunkai Zhang added a comment - [PATH V2 1/2] Allocator: optimize InkFreeList memory pool [PATH V2 2/2] Allocator: make InkFreeList memory pool configurable — ChangeLog: 1) By using alignment method, we can caclutate chunk address directly, instread of store a pointer back to InkChunkInfo in each allocation. 2) make the default "max_mem" to be 0, which means never try to reclaim memory in any time.
          Hide
          Leif Hedstrom added a comment -

          At a minimum, make the default "max_mem" be 0, which means never, ever, try to use this new reclaim code. Having a configure option would be cool too. I don't want a new config, with some arbitrary default (what did you expect to set max_mem to?), which the user misconfigures or don't configure at all, and we get bizarre behavior.

          Show
          Leif Hedstrom added a comment - At a minimum, make the default "max_mem" be 0, which means never, ever, try to use this new reclaim code. Having a configure option would be cool too. I don't want a new config, with some arbitrary default (what did you expect to set max_mem to?), which the user misconfigures or don't configure at all, and we get bizarre behavior.
          Hide
          John Plevyak added a comment -

          I agree. We should land this initially as a compile time option in the dev
          branch to get wider production time on it before moving it to default.

          The main reason is that it is invasive and complicated, particularly in the
          way it will interact with the VM system and it would be nice to see how it
          responds in a variety of environments.

          If it is much better than TCMalloc, then perhaps we should package it up in
          a more general form as well.

          Was the design based on another allocator/paper? Any references?

          john

          Show
          John Plevyak added a comment - I agree. We should land this initially as a compile time option in the dev branch to get wider production time on it before moving it to default. The main reason is that it is invasive and complicated, particularly in the way it will interact with the VM system and it would be nice to see how it responds in a variety of environments. If it is much better than TCMalloc, then perhaps we should package it up in a more general form as well. Was the design based on another allocator/paper? Any references? john
          Hide
          Leif Hedstrom added a comment -

          I haven't had a chance to review the patch (yet), but a few comments / concerns:

          1) I understand you want to reclaim the memory, to use it for another (non-ATS) process. I guess that's reasonable, but bear in mind that bad things can happen here. Such as, if you need 10G RAM to handle certain load, reclaim 4G of it to use for some other process, if you now see that same load again, you might not be able to allocate that full 10G again (at least not without going on swap).

          2) I also see / hear about problems where the claim is that we sort of leak, or don't use, the freelist as efficiently as possible. If that's the case, this patch seems like a bandaid (or duct tape) solution. I'm not opposing it per se, but doing this sort of garbage collection could then hide real, serious problems that we should otherwise fix.

          3) This becomes more difficult to configure. How do I know what to set the max memory to? How do I avoid the case where it constantly garbage collects off the freelist, just to cause it to malloc() new objects, and then garbage collect again? Such cycles could completely kill performance.

          At a minimum, I'd encourage that we make this behavior optional, and disabled by default. I'd be interested to hear from the Taoabao team and John about these concerns as well.

          Thanks!

          – Leif

          Show
          Leif Hedstrom added a comment - I haven't had a chance to review the patch (yet), but a few comments / concerns: 1) I understand you want to reclaim the memory, to use it for another (non-ATS) process. I guess that's reasonable, but bear in mind that bad things can happen here. Such as, if you need 10G RAM to handle certain load, reclaim 4G of it to use for some other process, if you now see that same load again, you might not be able to allocate that full 10G again (at least not without going on swap). 2) I also see / hear about problems where the claim is that we sort of leak, or don't use, the freelist as efficiently as possible. If that's the case, this patch seems like a bandaid (or duct tape) solution. I'm not opposing it per se, but doing this sort of garbage collection could then hide real, serious problems that we should otherwise fix. 3) This becomes more difficult to configure. How do I know what to set the max memory to? How do I avoid the case where it constantly garbage collects off the freelist, just to cause it to malloc() new objects, and then garbage collect again? Such cycles could completely kill performance. At a minimum, I'd encourage that we make this behavior optional, and disabled by default. I'd be interested to hear from the Taoabao team and John about these concerns as well. Thanks! – Leif
          Hide
          Yunkai Zhang added a comment - - edited

          John Plevyak

          InkThreadCache::status is an inner status(maybe the variable name is not so good, any suggestion?), need not to be listed, it represent the status(state) of allocator: Malloc-ing or Free-ing, I use it as an simple state machine - executing refresh_average_info() to calculate the minimum of free memory only when the status change from Malloc-ing to Free-ing. I will add more comments for it in the code.

          I have compare TS original allocator/JEMalloc/TCMalloc/LocklessMalloc before I optimize TS InkFreeList allocator, the result is that TS orginal allocator is faster than JEMalloc/TCMalloc/LocklessMalloc, this because JEMalloc/TCMalloc/LocklessMalloc is designed for general perpose.

          But my patches learn from those excellent general allocators, they are similar in some aspect.

          Show
          Yunkai Zhang added a comment - - edited John Plevyak InkThreadCache::status is an inner status(maybe the variable name is not so good, any suggestion?), need not to be listed, it represent the status(state) of allocator: Malloc-ing or Free-ing, I use it as an simple state machine - executing refresh_average_info() to calculate the minimum of free memory only when the status change from Malloc-ing to Free-ing. I will add more comments for it in the code. I have compare TS original allocator/JEMalloc/TCMalloc/LocklessMalloc before I optimize TS InkFreeList allocator, the result is that TS orginal allocator is faster than JEMalloc/TCMalloc/LocklessMalloc, this because JEMalloc/TCMalloc/LocklessMalloc is designed for general perpose . But my patches learn from those excellent general allocators, they are similar in some aspect.
          Hide
          John Plevyak added a comment -

          Some of the volatile variables are not listed as such (e.g. InkThreadCache::status).

          Also, what is the purpose of this status field and how is it updated? It is set in ink_freelist_new to 0 via simple assignment, then tested/assigned via a cas in ink_freelist_free. Some comments, or documentation would be nice.

          Have you tested this against the default memory allocator and TCMalloc?

          This seems to be doing something similar to TCMalloc and that code has been extensively tested.

          Show
          John Plevyak added a comment - Some of the volatile variables are not listed as such (e.g. InkThreadCache::status). Also, what is the purpose of this status field and how is it updated? It is set in ink_freelist_new to 0 via simple assignment, then tested/assigned via a cas in ink_freelist_free. Some comments, or documentation would be nice. Have you tested this against the default memory allocator and TCMalloc? This seems to be doing something similar to TCMalloc and that code has been extensively tested.
          Hide
          Yunkai Zhang added a comment - - edited

          Igor

          When allocator try to reclaim memory, it will harm performance. By setting appropriate max_mem/reclaim_factor/max_overage(they are configurable variables in recoreds.config file), we can let allocator running on the state that need not to reclaim memory at most time.

          In fact there are two conditions will lead to memory reclaiming:

          1) memory used by allocator exceed max_mem
          In this condition, it will always reclaim memory until memory size less than max_mem, this reclaiming will only slowdown the requests which lead to the memory beyond max_mem, but will not affect other requests.

          2) idle memory in allocator exceed chunck_size for max_overage times (see patch's commit log)
          When allocator has so many idle memory, it means that TS is not so busy, so the reclaiming in this condition will not harm performance.

          Show
          Yunkai Zhang added a comment - - edited Igor When allocator try to reclaim memory, it will harm performance. By setting appropriate max_mem/reclaim_factor/max_overage(they are configurable variables in recoreds.config file), we can let allocator running on the state that need not to reclaim memory at most time. In fact there are two conditions will lead to memory reclaiming: 1) memory used by allocator exceed max_mem In this condition, it will always reclaim memory until memory size less than max_mem, this reclaiming will only slowdown the requests which lead to the memory beyond max_mem, but will not affect other requests. 2) idle memory in allocator exceed chunck_size for max_overage times (see patch's commit log) When allocator has so many idle memory, it means that TS is not so busy, so the reclaiming in this condition will not harm performance.
          Hide
          Igor Galić added a comment -

          Yunkai Zhang can you explain what exactly you mean by "and will not harm the performance at most time"

          Show
          Igor Galić added a comment - Yunkai Zhang can you explain what exactly you mean by "and will not harm the performance at most time"
          Hide
          Yunkai Zhang added a comment - - edited

          see the attachment for my patchset.

          Show
          Yunkai Zhang added a comment - - edited see the attachment for my patchset.
          Hide
          Yunkai Zhang added a comment -

          Hi guys, I have developed a patchset to solve this issue. By using this patchset, the total memory used by TS have been controlled and will not harm the performance at most time.

          Show
          Yunkai Zhang added a comment - Hi guys, I have developed a patchset to solve this issue. By using this patchset, the total memory used by TS have been controlled and will not harm the performance at most time.
          Hide
          Zhao Yongming added a comment -

          not yet, we are still working on another solution, hopes we can submit for review in next week or soon.

          Show
          Zhao Yongming added a comment - not yet, we are still working on another solution, hopes we can submit for review in next week or soon.
          Hide
          samahee added a comment -

          it it fixed?

          Show
          samahee added a comment - it it fixed?
          Hide
          Leif Hedstrom added a comment -

          Moving this out to 3.3.0, please move back to 3.1.4 if this will be worked on soon. Also, take a look at what can be closed here please!

          Show
          Leif Hedstrom added a comment - Moving this out to 3.3.0, please move back to 3.1.4 if this will be worked on soon . Also, take a look at what can be closed here please!
          Hide
          Leif Hedstrom added a comment -

          Moved to 3.1.4, please move bugs back to 3.1.3, which you will work on in the next 2 weeks.

          Show
          Leif Hedstrom added a comment - Moved to 3.1.4, please move bugs back to 3.1.3, which you will work on in the next 2 weeks.
          Hide
          Zhao Yongming added a comment -

          the memory waste in TS is mostly because the Ramcache:
          1, it is counted by really used memory.
          2, it will hold the whole block of memory from free to OS, the old glibc memory management issue.
          3, ramcache use the cache reader buffer, and the buffer is allocated from anywhere in the whole memory adress

          all those make TS waste much more memory than Ramcache is configured. it will eat all you memory at then end.

          what we want to do:
          1, limit ramcache memory by allocator
          2, bound ram lur list and freelist
          3, freelist will be split by size
          4, split ramcache memory and cache memory

          the codes will be ready in days

          Show
          Zhao Yongming added a comment - the memory waste in TS is mostly because the Ramcache: 1, it is counted by really used memory. 2, it will hold the whole block of memory from free to OS, the old glibc memory management issue. 3, ramcache use the cache reader buffer, and the buffer is allocated from anywhere in the whole memory adress all those make TS waste much more memory than Ramcache is configured. it will eat all you memory at then end. what we want to do: 1, limit ramcache memory by allocator 2, bound ram lur list and freelist 3, freelist will be split by size 4, split ramcache memory and cache memory the codes will be ready in days
          Hide
          Zhao Yongming added a comment -

          This is the result after mohan_zl's patch that use Da & rbtree to reduce the mem alignemnt memory, 1.8G for my env.

          Show
          Zhao Yongming added a comment - This is the result after mohan_zl's patch that use Da & rbtree to reduce the mem alignemnt memory, 1.8G for my env.
          Hide
          Zhao Yongming added a comment -

          in this calc file, I have summary up all the memory usage that TS used, and the most heavy wasted memory lived in CacheBuf+IoBuf & CacheBuf alignment.
          we'd like to reduce the ~4G memory and hopes we can put that into ramcache and get better cache performance.

          Show
          Zhao Yongming added a comment - in this calc file, I have summary up all the memory usage that TS used, and the most heavy wasted memory lived in CacheBuf+IoBuf & CacheBuf alignment. we'd like to reduce the ~4G memory and hopes we can put that into ramcache and get better cache performance.
          Hide
          Zhao Yongming added a comment -

          we have made some enhancement and the result looks like the following, and we are still tracking for bugs and this is the first try, we will submit then codes after it stable for our usage.

          [Nov 19 11:25:42.351] Server {0x4257a940} WARNING: Document 644CE0FF truncated at 10485040 of 21289520, missing fragment 9DD38ED0
               allocated      |        in-use      | type size  |   free list name
          --------------------|--------------------|------------|----------------------------------
                     67108864 |                  0 |    2097152 | memory/cacheBufAllocator[14]
                     33554432 |                  0 |    1048576 | memory/cacheBufAllocator[13]
                     33554432 |            1572864 |     524288 | memory/cacheBufAllocator[12]
                     33554432 |            6029312 |     262144 | memory/cacheBufAllocator[11]
                     37748736 |           17301504 |     131072 | memory/cacheBufAllocator[10]
                     35651584 |           32964608 |      65536 | memory/cacheBufAllocator[9]
                     70254592 |           64782336 |      32768 | memory/cacheBufAllocator[8]
                     55574528 |           46366720 |      16384 | memory/cacheBufAllocator[7]
                     28573696 |           22749184 |       8192 | memory/cacheBufAllocator[6]
                     11534336 |            8888320 |       4096 | memory/cacheBufAllocator[5]
                            0 |                  0 |       2048 | memory/cacheBufAllocator[4]
                            0 |                  0 |       1024 | memory/cacheBufAllocator[3]
                            0 |                  0 |        512 | memory/cacheBufAllocator[2]
                            0 |                  0 |        256 | memory/cacheBufAllocator[1]
                            0 |                  0 |        128 | memory/cacheBufAllocator[0]
                      1572864 |             997376 |       2048 | memory/hdrStrHeap
                      2621440 |            1466368 |       2048 | memory/hdrHeap
                       262144 |             203776 |       1024 | memory/ArenaBlock
               dallocated     |        in-use      | type size  |   free list name
          --------------------|--------------------|------------|----------------------------------
                            0 |                  0 |    2097152 | memory/ioBufAllocator[14]
                            0 |                  0 |    1048576 | memory/ioBufAllocator[13]
                            0 |                  0 |     524288 | memory/ioBufAllocator[12]
                            0 |                  0 |     262144 | memory/ioBufAllocator[11]
                            0 |                  0 |     131072 | memory/ioBufAllocator[10]
                            0 |                  0 |      65536 | memory/ioBufAllocator[9]
                      7340032 |            6455296 |      32768 | memory/ioBufAllocator[8]
                      1048576 |             720896 |      16384 | memory/ioBufAllocator[7]
                      1310720 |            1048576 |       8192 | memory/ioBufAllocator[6]
                      1572864 |             929792 |       4096 | memory/ioBufAllocator[5]
                            0 |                  0 |       2048 | memory/ioBufAllocator[4]
                            0 |                  0 |       1024 | memory/ioBufAllocator[3]
                            0 |                  0 |        512 | memory/ioBufAllocator[2]
                            0 |                  0 |        256 | memory/ioBufAllocator[1]
                            0 |                  0 |        128 | memory/ioBufAllocator[0]
                            0 |                  0 |        592 | memory/ICPRequestCont_allocator
                            0 |                  0 |        128 | memory/ICPPeerReadContAllocator
                            0 |                  0 |        432 | memory/PeerReadDataAllocator
                            0 |                  0 |         32 | memory/MIMEFieldSDKHandle
                            0 |                  0 |        256 | memory/INKVConnAllocator
                            0 |                  0 |        112 | memory/INKContAllocator
                            0 |                  0 |         32 | memory/apiHookAllocator
                            0 |                  0 |        304 | memory/FetchSMAllocator
                            0 |                  0 |         80 | memory/prefetchLockHandlerAllocator
                            0 |                  0 |        176 | memory/PrefetchBlasterAllocator
                            0 |                  0 |         80 | memory/prefetchUrlBlaster
                            0 |                  0 |         96 | memory/blasterUrlList
                            0 |                  0 |         96 | memory/prefetchUrlEntryAllocator
                            0 |                  0 |        128 | memory/socksProxyAllocator
                            0 |                  0 |        160 | memory/ObjectReloadCont
                       155648 |             120992 |        608 | memory/httpClientSessionAllocator
                        26624 |               1456 |        208 | memory/httpServerSessionAllocator
                      2531328 |            1967712 |       9888 | memory/httpSMAllocator
                            0 |                  0 |         32 | memory/CacheLookupHttpConfigAllocator
                            0 |                  0 |       9936 | memory/httpUpdateSMAllocator
                            0 |                  0 |        128 | memory/RemapPluginsAlloc
                            0 |                  0 |        544 | memory/HCSMAllocator
                            0 |                  0 |         48 | memory/VCEntryAllocator
                            0 |                  0 |         96 | memory/HCEntryAllocator
                            0 |                  0 |         80 | memory/HCHandlerAllocator
                            0 |                  0 |         48 | memory/CongestRequestParamAllocator
                            0 |                  0 |        128 | memory/CongestionDBContAllocator
                        53248 |              29328 |        208 | memory/httpCacheAltAllocator
                            0 |                  0 |        128 | memory/OneWayTunnelAllocator
                       157696 |               1232 |       1232 | memory/hostDBContAllocator
                            0 |                  0 |      17040 | memory/dnsBufAllocator
                            0 |                  0 |       1264 | memory/dnsEntryAllocator
                            0 |                  0 |         16 | memory/DNSRequestDataAllocator
                            0 |                  0 |       1072 | memory/SRVAllocator
                            0 |                  0 |         48 | memory/ClusterVConnectionCache::Entry
                            0 |                  0 |        560 | memory/cacheContAllocator
                            0 |                  0 |        112 | memory/inControlAllocator
                            0 |                  0 |        128 | memory/outControlAllocator
                            0 |                  0 |         32 | memory/byteBankAllocator
                            0 |                  0 |        592 | memory/clusterVCAllocator
                            0 |                  0 |         48 | memory/evacuationKey
                            0 |                  0 |         64 | memory/cacheRemoveCont
                       258048 |             183168 |         96 | memory/evacuationBlock
                      3461120 |            2541760 |       1040 | memory/cacheVConnection
                       491520 |             337440 |        160 | memory/openDirEntry
                            0 |                  0 |         64 | memory/RamCacheLRUEntry
                      2297856 |            2290272 |         96 | memory/RamCacheCLFUSEntry
                       507904 |             489056 |        992 | memory/netVCAllocator
                            0 |                  0 |        128 | memory/udpReadContAllocator
                            0 |                  0 |        144 | memory/udpWorkContinuationAllocator
                            0 |                  0 |        160 | memory/udpPacketAllocator
                            0 |                  0 |        320 | memory/socksAllocator
                            0 |                  0 |       1120 | memory/sslNetVCAllocator
                            0 |                  0 |        128 | memory/UDPIOEventAllocator
                        49152 |              36352 |         64 | memory/ioBlockAllocator
                       620544 |             528576 |         48 | memory/ioDataAllocator
                        92160 |              49440 |        240 | memory/ioAllocator
                      1992704 |            1973104 |        112 | memory/mutexAllocator
                        98304 |              75264 |         96 | memory/eventAllocator
          
          Show
          Zhao Yongming added a comment - we have made some enhancement and the result looks like the following, and we are still tracking for bugs and this is the first try, we will submit then codes after it stable for our usage. [Nov 19 11:25:42.351] Server {0x4257a940} WARNING: Document 644CE0FF truncated at 10485040 of 21289520, missing fragment 9DD38ED0 allocated | in-use | type size | free list name --------------------|--------------------|------------|---------------------------------- 67108864 | 0 | 2097152 | memory/cacheBufAllocator[14] 33554432 | 0 | 1048576 | memory/cacheBufAllocator[13] 33554432 | 1572864 | 524288 | memory/cacheBufAllocator[12] 33554432 | 6029312 | 262144 | memory/cacheBufAllocator[11] 37748736 | 17301504 | 131072 | memory/cacheBufAllocator[10] 35651584 | 32964608 | 65536 | memory/cacheBufAllocator[9] 70254592 | 64782336 | 32768 | memory/cacheBufAllocator[8] 55574528 | 46366720 | 16384 | memory/cacheBufAllocator[7] 28573696 | 22749184 | 8192 | memory/cacheBufAllocator[6] 11534336 | 8888320 | 4096 | memory/cacheBufAllocator[5] 0 | 0 | 2048 | memory/cacheBufAllocator[4] 0 | 0 | 1024 | memory/cacheBufAllocator[3] 0 | 0 | 512 | memory/cacheBufAllocator[2] 0 | 0 | 256 | memory/cacheBufAllocator[1] 0 | 0 | 128 | memory/cacheBufAllocator[0] 1572864 | 997376 | 2048 | memory/hdrStrHeap 2621440 | 1466368 | 2048 | memory/hdrHeap 262144 | 203776 | 1024 | memory/ArenaBlock dallocated | in-use | type size | free list name --------------------|--------------------|------------|---------------------------------- 0 | 0 | 2097152 | memory/ioBufAllocator[14] 0 | 0 | 1048576 | memory/ioBufAllocator[13] 0 | 0 | 524288 | memory/ioBufAllocator[12] 0 | 0 | 262144 | memory/ioBufAllocator[11] 0 | 0 | 131072 | memory/ioBufAllocator[10] 0 | 0 | 65536 | memory/ioBufAllocator[9] 7340032 | 6455296 | 32768 | memory/ioBufAllocator[8] 1048576 | 720896 | 16384 | memory/ioBufAllocator[7] 1310720 | 1048576 | 8192 | memory/ioBufAllocator[6] 1572864 | 929792 | 4096 | memory/ioBufAllocator[5] 0 | 0 | 2048 | memory/ioBufAllocator[4] 0 | 0 | 1024 | memory/ioBufAllocator[3] 0 | 0 | 512 | memory/ioBufAllocator[2] 0 | 0 | 256 | memory/ioBufAllocator[1] 0 | 0 | 128 | memory/ioBufAllocator[0] 0 | 0 | 592 | memory/ICPRequestCont_allocator 0 | 0 | 128 | memory/ICPPeerReadContAllocator 0 | 0 | 432 | memory/PeerReadDataAllocator 0 | 0 | 32 | memory/MIMEFieldSDKHandle 0 | 0 | 256 | memory/INKVConnAllocator 0 | 0 | 112 | memory/INKContAllocator 0 | 0 | 32 | memory/apiHookAllocator 0 | 0 | 304 | memory/FetchSMAllocator 0 | 0 | 80 | memory/prefetchLockHandlerAllocator 0 | 0 | 176 | memory/PrefetchBlasterAllocator 0 | 0 | 80 | memory/prefetchUrlBlaster 0 | 0 | 96 | memory/blasterUrlList 0 | 0 | 96 | memory/prefetchUrlEntryAllocator 0 | 0 | 128 | memory/socksProxyAllocator 0 | 0 | 160 | memory/ObjectReloadCont 155648 | 120992 | 608 | memory/httpClientSessionAllocator 26624 | 1456 | 208 | memory/httpServerSessionAllocator 2531328 | 1967712 | 9888 | memory/httpSMAllocator 0 | 0 | 32 | memory/CacheLookupHttpConfigAllocator 0 | 0 | 9936 | memory/httpUpdateSMAllocator 0 | 0 | 128 | memory/RemapPluginsAlloc 0 | 0 | 544 | memory/HCSMAllocator 0 | 0 | 48 | memory/VCEntryAllocator 0 | 0 | 96 | memory/HCEntryAllocator 0 | 0 | 80 | memory/HCHandlerAllocator 0 | 0 | 48 | memory/CongestRequestParamAllocator 0 | 0 | 128 | memory/CongestionDBContAllocator 53248 | 29328 | 208 | memory/httpCacheAltAllocator 0 | 0 | 128 | memory/OneWayTunnelAllocator 157696 | 1232 | 1232 | memory/hostDBContAllocator 0 | 0 | 17040 | memory/dnsBufAllocator 0 | 0 | 1264 | memory/dnsEntryAllocator 0 | 0 | 16 | memory/DNSRequestDataAllocator 0 | 0 | 1072 | memory/SRVAllocator 0 | 0 | 48 | memory/ClusterVConnectionCache::Entry 0 | 0 | 560 | memory/cacheContAllocator 0 | 0 | 112 | memory/inControlAllocator 0 | 0 | 128 | memory/outControlAllocator 0 | 0 | 32 | memory/byteBankAllocator 0 | 0 | 592 | memory/clusterVCAllocator 0 | 0 | 48 | memory/evacuationKey 0 | 0 | 64 | memory/cacheRemoveCont 258048 | 183168 | 96 | memory/evacuationBlock 3461120 | 2541760 | 1040 | memory/cacheVConnection 491520 | 337440 | 160 | memory/openDirEntry 0 | 0 | 64 | memory/RamCacheLRUEntry 2297856 | 2290272 | 96 | memory/RamCacheCLFUSEntry 507904 | 489056 | 992 | memory/netVCAllocator 0 | 0 | 128 | memory/udpReadContAllocator 0 | 0 | 144 | memory/udpWorkContinuationAllocator 0 | 0 | 160 | memory/udpPacketAllocator 0 | 0 | 320 | memory/socksAllocator 0 | 0 | 1120 | memory/sslNetVCAllocator 0 | 0 | 128 | memory/UDPIOEventAllocator 49152 | 36352 | 64 | memory/ioBlockAllocator 620544 | 528576 | 48 | memory/ioDataAllocator 92160 | 49440 | 240 | memory/ioAllocator 1992704 | 1973104 | 112 | memory/mutexAllocator 98304 | 75264 | 96 | memory/eventAllocator
          Hide
          Conan Wang added a comment -

          my reverse box: 3.0.1 on centos 5.4 64bit

          physics memory: 4G
          RAM cache: 1.8G
          DISK: 407 GB (raw device)
          average_object_size: 8000

               allocated      |        in-use      | type size  |   free list name
          --------------------|--------------------|------------|----------------------------------
                     67108864 |                  0 |    2097152 | memory/ioBufAllocator[14]
                   1744830464 |           76546048 |    1048576 | memory/ioBufAllocator[13]
                     33554432 |            4194304 |     524288 | memory/ioBufAllocator[12]
                     33554432 |            6553600 |     262144 | memory/ioBufAllocator[11]
                     37748736 |            8912896 |     131072 | memory/ioBufAllocator[10]
                     69206016 |           31850496 |      65536 | memory/ioBufAllocator[9]
                    529530880 |          365658112 |      32768 | memory/ioBufAllocator[8]
                    414187520 |          411942912 |      16384 | memory/ioBufAllocator[7]
                    898629632 |          898547712 |       8192 | memory/ioBufAllocator[6]
                     83361792 |           83161088 |       4096 | memory/ioBufAllocator[5]
                       262144 |                  0 |       2048 | memory/ioBufAllocator[4]
                       131072 |                  0 |       1024 | memory/ioBufAllocator[3]
                        65536 |                  0 |        512 | memory/ioBufAllocator[2]
                        32768 |                  0 |        256 | memory/ioBufAllocator[1]
                            0 |                  0 |        128 | memory/ioBufAllocator[0]
                            0 |                  0 |        576 | memory/ICPRequestCont_allocator
                            0 |                  0 |        112 | memory/ICPPeerReadContAllocator
                            0 |                  0 |        432 | memory/PeerReadDataAllocator
                         4096 |                160 |         32 | memory/MIMEFieldSDKHandle
                            0 |                  0 |        240 | memory/INKVConnAllocator
                        12288 |                 96 |         96 | memory/INKContAllocator
                         4096 |                 32 |         32 | memory/apiHookAllocator
                            0 |                  0 |        288 | memory/FetchSMAllocator
                            0 |                  0 |         80 | memory/prefetchLockHandlerAllocator
                            0 |                  0 |        176 | memory/PrefetchBlasterAllocator
                            0 |                  0 |         80 | memory/prefetchUrlBlaster
                            0 |                  0 |         96 | memory/blasterUrlList
                            0 |                  0 |         96 | memory/prefetchUrlEntryAllocator
                            0 |                  0 |        128 | memory/socksProxyAllocator
                            0 |                  0 |        144 | memory/ObjectReloadCont
                      1060864 |               2960 |        592 | memory/httpClientSessionAllocator
                        53248 |                416 |        208 | memory/httpServerSessionAllocator
                     17575936 |              19616 |       9808 | memory/httpSMAllocator
                            0 |                  0 |         32 | memory/CacheLookupHttpConfigAllocator
                            0 |                  0 |       9856 | memory/httpUpdateSMAllocator
                            0 |                  0 |        128 | memory/RemapPluginsAlloc
                            0 |                  0 |         48 | memory/CongestRequestParamAllocator
                            0 |                  0 |        128 | memory/CongestionDBContAllocator
                      7340032 |             264192 |       2048 | memory/hdrStrHeap
                      8650752 |             274432 |       2048 | memory/hdrHeap
          #
                        26624 |                  0 |        208 | memory/httpCacheAltAllocator
                            0 |                  0 |        112 | memory/OneWayTunnelAllocator
                       315392 |               1232 |       1232 | memory/hostDBContAllocator
                        68160 |              17040 |      17040 | memory/dnsBufAllocator
                       161792 |                  0 |       1264 | memory/dnsEntryAllocator
                            0 |                  0 |         16 | memory/DNSRequestDataAllocator
                            0 |                  0 |       1072 | memory/SRVAllocator
                            0 |                  0 |         48 | memory/ClusterVConnectionCache::Entry
                            0 |                  0 |        560 | memory/cacheContAllocator
                            0 |                  0 |        112 | memory/inControlAllocator
                            0 |                  0 |        112 | memory/outControlAllocator
                            0 |                  0 |         32 | memory/byteBankAllocator
                            0 |                  0 |        576 | memory/clusterVCAllocator
                            0 |                  0 |         48 | memory/evacuationKey
                         6144 |                  0 |         48 | memory/cacheRemoveCont
                       122880 |             118272 |         96 | memory/evacuationBlock
                      1374208 |            1013088 |        976 | memory/cacheVConnection
                       245760 |             137920 |        160 | memory/openDirEntry
                            0 |                  0 |         64 | memory/RamCacheLRUEntry
                     32059392 |           32056512 |         96 | memory/RamCacheCLFUSEntry
                      2949120 |             174720 |        960 | memory/netVCAllocator
                            0 |                  0 |        128 | memory/udpReadContAllocator
                            0 |                  0 |        128 | memory/udpWorkContinuationAllocator
                            0 |                  0 |        160 | memory/udpPacketAllocator
                            0 |                  0 |        304 | memory/socksAllocator
                            0 |                  0 |       1088 | memory/sslNetVCAllocator
                            0 |                  0 |        128 | memory/UDPIOEventAllocator
                       294912 |                512 |         64 | memory/ioBlockAllocator
                      8030208 |            8013648 |         48 | memory/ioDataAllocator
                       583680 |               1920 |        240 | memory/ioAllocator
                      1617920 |            1382560 |         80 | memory/mutexAllocator
                       368640 |             110592 |         96 | memory/eventAllocator
                      1703936 |               2048 |       1024 | memory/ArenaBlock
          
          
          top
            PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP DATA COMMAND                                                                                
          27172 nobody    15   0 5083m 3.1g 4344 S  2.0 81.2  67:21.32 1.8g 4.9g [ET_NET 0]  
          
          ps aux
          USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
          nobody   27172  4.9 81.1 5205836 3286380 ?     Sl   Oct29  67:24 /usr/local/bin/traffic_server -M -A,7:X
          
          Show
          Conan Wang added a comment - my reverse box: 3.0.1 on centos 5.4 64bit physics memory: 4G RAM cache: 1.8G DISK: 407 GB (raw device) average_object_size: 8000 allocated | in-use | type size | free list name --------------------|--------------------|------------|---------------------------------- 67108864 | 0 | 2097152 | memory/ioBufAllocator[14] 1744830464 | 76546048 | 1048576 | memory/ioBufAllocator[13] 33554432 | 4194304 | 524288 | memory/ioBufAllocator[12] 33554432 | 6553600 | 262144 | memory/ioBufAllocator[11] 37748736 | 8912896 | 131072 | memory/ioBufAllocator[10] 69206016 | 31850496 | 65536 | memory/ioBufAllocator[9] 529530880 | 365658112 | 32768 | memory/ioBufAllocator[8] 414187520 | 411942912 | 16384 | memory/ioBufAllocator[7] 898629632 | 898547712 | 8192 | memory/ioBufAllocator[6] 83361792 | 83161088 | 4096 | memory/ioBufAllocator[5] 262144 | 0 | 2048 | memory/ioBufAllocator[4] 131072 | 0 | 1024 | memory/ioBufAllocator[3] 65536 | 0 | 512 | memory/ioBufAllocator[2] 32768 | 0 | 256 | memory/ioBufAllocator[1] 0 | 0 | 128 | memory/ioBufAllocator[0] 0 | 0 | 576 | memory/ICPRequestCont_allocator 0 | 0 | 112 | memory/ICPPeerReadContAllocator 0 | 0 | 432 | memory/PeerReadDataAllocator 4096 | 160 | 32 | memory/MIMEFieldSDKHandle 0 | 0 | 240 | memory/INKVConnAllocator 12288 | 96 | 96 | memory/INKContAllocator 4096 | 32 | 32 | memory/apiHookAllocator 0 | 0 | 288 | memory/FetchSMAllocator 0 | 0 | 80 | memory/prefetchLockHandlerAllocator 0 | 0 | 176 | memory/PrefetchBlasterAllocator 0 | 0 | 80 | memory/prefetchUrlBlaster 0 | 0 | 96 | memory/blasterUrlList 0 | 0 | 96 | memory/prefetchUrlEntryAllocator 0 | 0 | 128 | memory/socksProxyAllocator 0 | 0 | 144 | memory/ObjectReloadCont 1060864 | 2960 | 592 | memory/httpClientSessionAllocator 53248 | 416 | 208 | memory/httpServerSessionAllocator 17575936 | 19616 | 9808 | memory/httpSMAllocator 0 | 0 | 32 | memory/CacheLookupHttpConfigAllocator 0 | 0 | 9856 | memory/httpUpdateSMAllocator 0 | 0 | 128 | memory/RemapPluginsAlloc 0 | 0 | 48 | memory/CongestRequestParamAllocator 0 | 0 | 128 | memory/CongestionDBContAllocator 7340032 | 264192 | 2048 | memory/hdrStrHeap 8650752 | 274432 | 2048 | memory/hdrHeap # 26624 | 0 | 208 | memory/httpCacheAltAllocator 0 | 0 | 112 | memory/OneWayTunnelAllocator 315392 | 1232 | 1232 | memory/hostDBContAllocator 68160 | 17040 | 17040 | memory/dnsBufAllocator 161792 | 0 | 1264 | memory/dnsEntryAllocator 0 | 0 | 16 | memory/DNSRequestDataAllocator 0 | 0 | 1072 | memory/SRVAllocator 0 | 0 | 48 | memory/ClusterVConnectionCache::Entry 0 | 0 | 560 | memory/cacheContAllocator 0 | 0 | 112 | memory/inControlAllocator 0 | 0 | 112 | memory/outControlAllocator 0 | 0 | 32 | memory/byteBankAllocator 0 | 0 | 576 | memory/clusterVCAllocator 0 | 0 | 48 | memory/evacuationKey 6144 | 0 | 48 | memory/cacheRemoveCont 122880 | 118272 | 96 | memory/evacuationBlock 1374208 | 1013088 | 976 | memory/cacheVConnection 245760 | 137920 | 160 | memory/openDirEntry 0 | 0 | 64 | memory/RamCacheLRUEntry 32059392 | 32056512 | 96 | memory/RamCacheCLFUSEntry 2949120 | 174720 | 960 | memory/netVCAllocator 0 | 0 | 128 | memory/udpReadContAllocator 0 | 0 | 128 | memory/udpWorkContinuationAllocator 0 | 0 | 160 | memory/udpPacketAllocator 0 | 0 | 304 | memory/socksAllocator 0 | 0 | 1088 | memory/sslNetVCAllocator 0 | 0 | 128 | memory/UDPIOEventAllocator 294912 | 512 | 64 | memory/ioBlockAllocator 8030208 | 8013648 | 48 | memory/ioDataAllocator 583680 | 1920 | 240 | memory/ioAllocator 1617920 | 1382560 | 80 | memory/mutexAllocator 368640 | 110592 | 96 | memory/eventAllocator 1703936 | 2048 | 1024 | memory/ArenaBlock top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP DATA COMMAND 27172 nobody 15 0 5083m 3.1g 4344 S 2.0 81.2 67:21.32 1.8g 4.9g [ET_NET 0] ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND nobody 27172 4.9 81.1 5205836 3286380 ? Sl Oct29 67:24 /usr/local/bin/traffic_server -M -A,7:X

            People

            • Assignee:
              Yunkai Zhang
              Reporter:
              Zhao Yongming
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development