Major Description
During our production testing of ATS master, I noticed that ATS memory keeps growing beyond ram cache size limits. Upon enabling the memory allocation dump, it looks like the ioBufAllocator objects keep growing endlessly. Talking to psudaemon and amc on irc, it looks like the issue might be caused due to NOT in-use proxy allocator (local thread) objects accumulating until they reach the per thread object count limit (250). This may be significant memory for large sized objects (e.g 1MB or 2MB). psudaemon recommended to try reverting the changes in TS-2592 that force to use Proxy allocator instead of global pool objects for ioBuffers. (Note that my production host has ram cache set to 6G and that is completely used, but, the ioBuffers just keep growing)
Here's the dump of memory allocations:
-bash-4.1$ sudo grep "ioBufAllocator\[13\]" traffic.out
0 | 0 | 1048576 | memory/ioBufAllocator[13]
1174405120 | 1151336448 | 1048576 | memory/ioBufAllocator[13]
4194304000 | 4186963968 | 1048576 | memory/ioBufAllocator[13]
1912602624 | 1908408320 | 1048576 | memory/ioBufAllocator[13]
771751936 | 759169024 | 1048576 | memory/ioBufAllocator[13]
1610612736 | 1608515584 | 1048576 | memory/ioBufAllocator[13]
1509949440 | 1484783616 | 1048576 | memory/ioBufAllocator[13]
3724541952 | 3697278976 | 1048576 | memory/ioBufAllocator[13]
2046820352 | 2023751680 | 1048576 | memory/ioBufAllocator[13]
2046820352 | 2036334592 | 1048576 | memory/ioBufAllocator[13]
2147483648 | 2128609280 | 1048576 | memory/ioBufAllocator[13]
2181038080 | 2172649472 | 1048576 | memory/ioBufAllocator[13]
2214592512 | 2204106752 | 1048576 | memory/ioBufAllocator[13]
2281701376 | 2253389824 | 1048576 | memory/ioBufAllocator[13]
2315255808 | 2287992832 | 1048576 | memory/ioBufAllocator[13]
2348810240 | 2316304384 | 1048576 | memory/ioBufAllocator[13]
2348810240 | 2336227328 | 1048576 | memory/ioBufAllocator[13]
2382364672 | 2380267520 | 1048576 | memory/ioBufAllocator[13]
2449473536 | 2424307712 | 1048576 | memory/ioBufAllocator[13]
2449473536 | 2447376384 | 1048576 | memory/ioBufAllocator[13]
2483027968 | 2474639360 | 1048576 | memory/ioBufAllocator[13]
2516582400 | 2507145216 | 1048576 | memory/ioBufAllocator[13]
2550136832 | 2543845376 | 1048576 | memory/ioBufAllocator[13]
2617245696 | 2599419904 | 1048576 | memory/ioBufAllocator[13]
2650800128 | 2626682880 | 1048576 | memory/ioBufAllocator[13]
2684354560 | 2659188736 | 1048576 | memory/ioBufAllocator[13]
2684354560 | 2675965952 | 1048576 | memory/ioBufAllocator[13]
2717908992 | 2690646016 | 1048576 | memory/ioBufAllocator[13]
2717908992 | 2713714688 | 1048576 | memory/ioBufAllocator[13]
2785017856 | 2758803456 | 1048576 | memory/ioBufAllocator[13]
2818572288 | 2792357888 | 1048576 | memory/ioBufAllocator[13]
2852126720 | 2823815168 | 1048576 | memory/ioBufAllocator[13]
2885681152 | 2882535424 | 1048576 | memory/ioBufAllocator[13]
2952790016 | 2921332736 | 1048576 | memory/ioBufAllocator[13]
2986344448 | 2959081472 | 1048576 | memory/ioBufAllocator[13]
2986344448 | 2985295872 | 1048576 | memory/ioBufAllocator[13]
3019898880 | 3014656000 | 1048576 | memory/ioBufAllocator[13]
3087007744 | 3067084800 | 1048576 | memory/ioBufAllocator[13]
3120562176 | 3104833536 | 1048576 | memory/ioBufAllocator[13]
3154116608 | 3146776576 | 1048576 | memory/ioBufAllocator[13]
3187671040 | 3186622464 | 1048576 | memory/ioBufAllocator[13]
3221225472 | 3220176896 | 1048576 | memory/ioBufAllocator[13]
3288334336 | 3268411392 | 1048576 | memory/ioBufAllocator[13]
3321888768 | 3292528640 | 1048576 | memory/ioBufAllocator[13]
3355443200 | 3344957440 | 1048576 | memory/ioBufAllocator[13]
3388997632 | 3364880384 | 1048576 | memory/ioBufAllocator[13]
3388997632 | 3387949056 | 1048576 | memory/ioBufAllocator[13]
3422552064 | 3404726272 | 1048576 | memory/ioBufAllocator[13]
3456106496 | 3423600640 | 1048576 | memory/ioBufAllocator[13]
Complete memory dump at some point:
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
201326592 | 174063616 | 2097152 | memory/ioBufAllocator[14]
3623878656 | 3597664256 | 1048576 | memory/ioBufAllocator[13]
1811939328 | 1811939328 | 524288 | memory/ioBufAllocator[12]
1400897536 | 1400635392 | 262144 | memory/ioBufAllocator[11]
1283457024 | 1281490944 | 131072 | memory/ioBufAllocator[10]
1663041536 | 1542127616 | 65536 | memory/ioBufAllocator[9]
1693450240 | 1058897920 | 32768 | memory/ioBufAllocator[8]
725614592 | 635879424 | 16384 | memory/ioBufAllocator[7]
398196736 | 357244928 | 8192 | memory/ioBufAllocator[6]
856686592 | 833167360 | 4096 | memory/ioBufAllocator[5]
0 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 61440 | 1024 | memory/ioBufAllocator[3]
65536 | 24576 | 512 | memory/ioBufAllocator[2]
1638400 | 1135616 | 256 | memory/ioBufAllocator[1]
1081344 | 643072 | 128 | memory/ioBufAllocator[0]
4325376 | 250368 | 512 | memory/FetchSMAllocator
0 | 0 | 592 | memory/ICPRequestCont_allocator
0 | 0 | 112 | memory/ICPPeerReadContAllocator
0 | 0 | 432 | memory/PeerReadDataAllocator
4096 | 0 | 32 | memory/MIMEFieldSDKHandle
0 | 0 | 240 | memory/INKVConnAllocator
12288 | 1248 | 96 | memory/INKContAllocator
4096 | 32 | 32 | memory/apiHookAllocator
0 | 0 | 96 | memory/prefetchLockHandlerAllocator
0 | 0 | 320 | memory/PrefetchBlasterAllocator
0 | 0 | 208 | memory/prefetchUrlBlaster
0 | 0 | 224 | memory/blasterUrlList
0 | 0 | 192 | memory/prefetchUrlEntryAllocator
0 | 0 | 128 | memory/socksProxyAllocator
0 | 0 | 144 | memory/ObjectReloadCont
16625664 | 13732048 | 656 | memory/httpClientSessionAllocator
56150016 | 1593072 | 7696 | memory/httpSMAllocator
200704 | 62720 | 224 | memory/httpServerSessionAllocator
0 | 0 | 48 | memory/CacheLookupHttpConfigAllocator
0 | 0 | 7744 | memory/httpUpdateSMAllocator
1892352 | 109312 | 224 | memory/spdyRequestAllocator
1794048 | 1698432 | 192 | memory/spdyClientSessionAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 48 | memory/CongestRequestParamAllocator
0 | 0 | 144 | memory/CongestionDBContAllocator
360448 | 1792 | 256 | memory/httpCacheAltAllocator
41943040 | 20996096 | 2048 | memory/hdrStrHeap
58982400 | 23791616 | 2048 | memory/hdrHeap
0 | 0 | 112 | memory/OneWayTunnelAllocator
589824 | 2304 | 2304 | memory/hostDBContAllocator
203136 | 33856 | 33856 | memory/dnsBufAllocator
327680 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 | memory/DNSRequestDataAllocator
0 | 0 | 48 | memory/ClusterVConnectionCache::Entry
0 | 0 | 576 | memory/cacheContAllocator
0 | 0 | 112 | memory/inControlAllocator
0 | 0 | 112 | memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 | memory/clusterVCAllocator
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
208896 | 207552 | 96 | memory/evacuationBlock
9621504 | 6307616 | 928 | memory/cacheVConnection
1474560 | 941280 | 160 | memory/openDirEntry
0 | 0 | 96 | memory/RamCacheCLFUSEntry
8986624 | 8785728 | 64 | memory/RamCacheLRUEntry
69396480 | 68240160 | 720 | memory/sslNetVCAllocator
0 | 0 | 128 | memory/UDPIOEventAllocator
13473792 | 11788192 | 688 | memory/netVCAllocator
0 | 0 | 128 | memory/udpReadContAllocator
0 | 0 | 160 | memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
14876672 | 12889408 | 64 | memory/ioBlockAllocator
17006592 | 15877632 | 48 | memory/ioDataAllocator
34836480 | 27895920 | 240 | memory/ioAllocator
11970560 | 11386880 | 80 | memory/mutexAllocator
3170304 | 1586976 | 96 | memory/eventAllocator
4456448 | 159744 | 1024 | memory/ArenaBlock
top command showing traffic_server taking 15g RES mem:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6274 nobody 20 0 19.9g 15g 8428 S 480.3 65.2 1798:44 /home/y/bin/traffic_server -M --httpport 80:fd=7,81:fd=8,82:fd=9,83:fd=10,84:fd=11,85:fd=12,90:fd=13,80:fd=14:ipv6,81:fd=15
Attachments
Issue Links
- is related to
-
TS-2592 ioBufAllocator not using proxyAllocator
-
- Closed
-