Uploaded image for project: 'Traffic Server'
  1. Traffic Server
  2. TS-4266

ATS memory statistics shows that memory utilization is doubled after “traffic_ctl config reload”



    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 6.2.0
    • Lua
    • None


      ATS memory statistics shows memory utilization is doubled after “traffic_ctl config reload”. We get “not enough memory” error in the subsequent attempt and “config reload” fails.

      ATS is configured with 100 map entries in remap.config, all share the same lua script.

      ATS is started: The memory information is..
      [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
      total kB 1416092 670256 663736

      After 1st Config reload:
      [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
      total kB 1932660 1128084 1121544

      After 2nd config reload: It had failed with error “not enough memory” and memory status as..
      [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
      total kB 2170756 1167808 1160836

      Error displayed in diags.log:
      [Mar 8 23:27:27.580] Server

      {0x2af92498b700} WARNING: Failed to create new instance for plugin /opt/trafficserver/libexec/trafficserver/tslua.so (not a TS_SUCCESS return)
      [Mar 8 23:27:27.580] Server {0x2af92498b700}

      WARNING: Could not add rule at line #3; Aborting!
      [Mar 8 23:27:27.580] Server

      {0x2af92498b700} WARNING: [ReverseProxy] Can't create new remap instance for plugin "/opt/trafficserver/libexec/trafficserver/tslua.so" - [ts_lua_add_module] luaL_loadfile /opt/trafficserver/etc/trafficserver/lua/process_remap.lua failed: not enough memory at line 3
      [Mar 8 23:27:27.580] Server {0x2af92498b700}

      WARNING: something failed during BuildTable() – check your remap plugins!
      [Mar 8 23:27:27.595] Server

      {0x2af92498b700} WARNING: failed to reload remap.config, not replacing!

      Lua VM memory size at that time ,ts.debug(FUNCTION..'Lua VM memory: '..collectgarbage("count"))
      [Mar 8 23:27:27.579] Server {0x2af92498b700}

      DIAG: (ts_lua) _init_(): Lua VM memory: 3629.7060546875

      This shows that Lua VMs are hitting the max capacity.

      I looked at the ts_lua code TSRemapDeleteInstance () [ts_lua.c ] and ts_lua_del_module() [ts_lua_util.c] which does cleaning of the lua memory for the instance. However the lua memory is not released and reused.
      So, I have added code to start the garbage collector in ts_lua_del_module() .
      ts_lua_del_module(ts_lua_instance_conf *conf, ts_lua_main_ctx *arr, int n)
      lua_replace(L, LUA_GLOBALSINDEX); /* L[GLOBAL] = EMPTY */

      lua_gc(L, LUA_GCCOLLECT, 0);


      return 0;
      This has improved the situation. However, I also added garbage collection in ts_lua_add_module() at the end. With these two additions, we have tested the code, the memory utilization is stable and we could do config reload at lest 100 times with the background load.




            kichan Shu Kit Francis Chan
            rkishore Rajendra Kishore Bonumahanti
            0 Vote for this issue
            6 Start watching this issue