Critical Description
After fixing the iobuffer leak in TS-3257, the iobuffers seem stable on v5.2.0, but, there still seems to be a slow memory leak. The RES memory from top shows 15g after running v5.2.0 in prod for more than 3 days, whereas the corresponding v5.0 host shows 10g after running for more than a week.
Below is the dump of iobuffers between the v5.2.0 and v5.0 host - as expected, most iobufs are lower than v5.0 host (since, the v5.0 host been running longer), except the 32k buffer (iobuf[8]). But, the leak doesn't seem to be explained by the difference in 32k buffers either, as it is not high enough to explain the 5g difference in total memory.
v5.2.0 host:
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
67108864 | 25165824 | 2097152 | memory/ioBufAllocator[14]
2013265920 | 1825570816 | 1048576 | memory/ioBufAllocator[13]
620756992 | 549978112 | 524288 | memory/ioBufAllocator[12]
780140544 | 593494016 | 262144 | memory/ioBufAllocator[11]
742391808 | 574619648 | 131072 | memory/ioBufAllocator[10]
901775360 | 735576064 | 65536 | memory/ioBufAllocator[9]
1189085184 | 1093304320 | 32768 | memory/ioBufAllocator[8]
474480640 | 348733440 | 16384 | memory/ioBufAllocator[7]
269221888 | 211320832 | 8192 | memory/ioBufAllocator[6]
156762112 | 142999552 | 4096 | memory/ioBufAllocator[5]
0 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 0 | 1024 | memory/ioBufAllocator[3]
65536 | 0 | 512 | memory/ioBufAllocator[2]
65536 | 256 | 256 | memory/ioBufAllocator[1]
16384 | 0 | 128 | memory/ioBufAllocator[0]
v.5.0.0 host:
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
134217728 | 31457280 | 2097152 | memory/ioBufAllocator[14]
2147483648 | 1843396608 | 1048576 | memory/ioBufAllocator[13]
788529152 | 608174080 | 524288 | memory/ioBufAllocator[12]
897581056 | 680525824 | 262144 | memory/ioBufAllocator[11]
796917760 | 660471808 | 131072 | memory/ioBufAllocator[10]
985661440 | 818479104 | 65536 | memory/ioBufAllocator[9]
873463808 | 677969920 | 32768 | memory/ioBufAllocator[8]
544735232 | 404439040 | 16384 | memory/ioBufAllocator[7]
310902784 | 237887488 | 8192 | memory/ioBufAllocator[6]
160956416 | 115515392 | 4096 | memory/ioBufAllocator[5]
0 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 2048 | 1024 | memory/ioBufAllocator[3]
65536 | 0 | 512 | memory/ioBufAllocator[2]
98304 | 50688 | 256 | memory/ioBufAllocator[1]