Major Description
ATS memory statistics shows memory utilization is doubled after “traffic_ctl config reload”. We get “not enough memory” error in the subsequent attempt and “config reload” fails.
ATS is configured with 100 map entries in remap.config, all share the same lua script.
ATS is started: The memory information is..
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB 1416092 670256 663736
After 1st Config reload:
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB 1932660 1128084 1121544
After 2nd config reload: It had failed with error “not enough memory” and memory status as..
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB 2170756 1167808 1160836
Error displayed in diags.log:
=======================
[Mar 8 23:27:27.580] Server
[Mar 8 23:27:27.580] Server {0x2af92498b700}
WARNING: Could not add rule at line #3; Aborting!
[Mar 8 23:27:27.580] Server
[Mar 8 23:27:27.580] Server {0x2af92498b700}
WARNING: something failed during BuildTable() – check your remap plugins!
[Mar 8 23:27:27.595] Server
Lua VM memory size at that time ,ts.debug(FUNCTION..'Lua VM memory: '..collectgarbage("count"))
[Mar 8 23:27:27.579] Server {0x2af92498b700}
DIAG: (ts_lua) _init_(): Lua VM memory: 3629.7060546875
This shows that Lua VMs are hitting the max capacity.
Solution:
=======
I looked at the ts_lua code TSRemapDeleteInstance () [ts_lua.c ] and ts_lua_del_module() [ts_lua_util.c] which does cleaning of the lua memory for the instance. However the lua memory is not released and reused.
So, I have added code to start the garbage collector in ts_lua_del_module() .
int
ts_lua_del_module(ts_lua_instance_conf *conf, ts_lua_main_ctx *arr, int n)
{
….
lua_newtable(L);
lua_replace(L, LUA_GLOBALSINDEX); /* L[GLOBAL] = EMPTY */
lua_gc(L, LUA_GCCOLLECT, 0);
TSMutexUnlock(arr[i].mutexp);
}
return 0;
}
This has improved the situation. However, I also added garbage collection in ts_lua_add_module() at the end. With these two additions, we have tested the code, the memory utilization is stable and we could do config reload at lest 100 times with the background load.