Description
(This is from a Y! Bugzilla ticket 1821593, adding it here. . Originally posted by Leif Hedstrom on 2008-03-19):
Currently you have to set static keep-alive idle timeouts in TS, e.g.
CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 8
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 30
even with epoll() in 1.17.x, this is difficult to configure, and put an appropriate timeout. The key here is that the
settings above need to assure that you stay below the max configured number of connections, e.g.:
CONFIG proxy.config.net.connections_throttle INT 75000
I'm suggesting that we add one (or two) new configuration options, and appropriate TS code support, to instead of
specifying timeouts, we specify connection limits for idle KA connections. For example:
CONFIG proxy.config.http.keep_alive_max_idle_connections_in INT 50000
CONFIG proxy.config.http_keep_alive_max_idle_connections_out INT 5000
(one still has to be careful to leave head-room for active connections here, in the example above, 20000 connections
could be active, which is a lot of traffic).
These would override the idle timeouts, so one could use the max_idle connections for incoming (client) connections,
and the idle timeouts for outgoing (origin) connections for instance.
The benefit here is that it makes configuration not only easier, but also a lot safer for many applications.
Attachments
Attachments
- ts153.diff
- 10 kB
- Bryan Call
Issue Links
Activity
Moving to 5.0.0 as per https://cwiki.apache.org/confluence/display/TS/New+Release+Processes
investigating the current timeouts here is the debug output of setting the timeouts on a single forward proxy requests:
[Nov 6 10:55:50.819] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=120000000000, for NetVC=0x7fa9a3855160 [Nov 6 10:55:50.819] Server {0x10e59b000} DEBUG: (socket) Set active timeout=900000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:50.819] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 120000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:50.819] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=30000000000, for NetVC=0x7fa9a3855160 [Nov 6 10:55:50.819] Server {0x10e59b000} DEBUG: (socket) read_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:50.865] Server {0x10e59b000} DEBUG: (socket) Cancel inactive timeout for NetVC=0x7fa9a3855160 [Nov 6 10:55:50.865] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=30000000000, for NetVC=0x7fa9a3850d20 [Nov 6 10:55:50.865] Server {0x10e59b000} DEBUG: (socket) Set active timeout=0, NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.065] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.602] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.602] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=30000000000, for NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.603] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=30000000000, for NetVC=0x7fa9a3855160 [Nov 6 10:55:51.603] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:51.603] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:51.604] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.604] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:51.604] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:51.802] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.802] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.802] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:51.802] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:51.993] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:51.993] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:51.993] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.046] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.046] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.046] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.059] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.059] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.059] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.059] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.069] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.069] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.069] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.238] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.238] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.238] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.238] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.281] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.281] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.281] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.281] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.309] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.309] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.309] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.430] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.430] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.430] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.448] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.448] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.448] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.448] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.448] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.448] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.490] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.522] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.522] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.522] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.522] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.522] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.522] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.525] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.525] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.525] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.637] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.637] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.637] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.645] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.645] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.645] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.645] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.649] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.649] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.649] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.682] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.682] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.682] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.686] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.700] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.700] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.700] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.700] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.770] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.770] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.770] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.772] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.772] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.772] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.772] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.772] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.772] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.773] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.773] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.773] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.773] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.773] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.773] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.773] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.842] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.843] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.843] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.847] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.847] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.847] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.847] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.852] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.852] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.852] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.852] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.889] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.889] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.889] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.898] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.898] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.898] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.899] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.899] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.900] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.925] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.925] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.925] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.938] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.938] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.938] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.971] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.971] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.971] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.988] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:52.988] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:52.988] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.010] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.010] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.010] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.036] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.036] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.036] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.040] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.040] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.040] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.052] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.052] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.052] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.069] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.070] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.070] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.072] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.073] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.073] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.074] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.074] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.074] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.074] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.074] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.078] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.078] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.078] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.078] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.079] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.079] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.079] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.079] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.079] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.081] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.081] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.081] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.225] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.226] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.226] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.261] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.262] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.262] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.272] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.272] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.272] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.272] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.272] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.278] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.278] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.278] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.279] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.281] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.281] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.281] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.281] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.281] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.281] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.284] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.470] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.470] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.471] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.474] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.476] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.476] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.476] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.532] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.533] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.533] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.552] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.552] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.552] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.552] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.572] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.573] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.573] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.573] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.573] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.685] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.686] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.686] Server {0x10e59b000} DEBUG: (socket) write_disable updating inactivity_at 0, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.702] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.702] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.702] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=120000000000, for NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.702] Server {0x10e59b000} DEBUG: (socket) net_activity updating inactivity 30000000000, NetVC=0x7fa9a3855160 [Nov 6 10:55:53.702] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=115000000000, for NetVC=0x7fa9a3855160 [Nov 6 10:55:53.702] Server {0x10e59b000} DEBUG: (socket) Cancel active timeout for NetVC=0x7fa9a3855160 [Nov 6 10:55:53.704] Server {0x10e59b000} DEBUG: (socket) Set inactive timeout=120000000000, for NetVC=0x7fa9a3850d20 [Nov 6 10:55:53.704] Server {0x10e59b000} DEBUG: (socket) Set active timeout=0, NetVC=0x7fa9a3850d20
Production "debug" output running with 20 threads and the limit set to 500. Sill need to make changes to the configuration to make it more user friendly. LRU size is before is starts to close out the keep-alive connections. max: 500 is how many connection that are permitted in the LRU, so to ats will close down connections to hit that value. total: is the total number of origin and client connections in that NetHandler.
[Nov 8 22:46:05.773] Server {0x2b8803f4e700} ERROR: keep alive lru size: 506 max: 500 total: 571 [Nov 8 22:46:05.787] Server {0x2b8801d823a0} ERROR: keep alive lru size: 513 max: 500 total: 568 [Nov 8 22:46:05.788] Server {0x2b8808504700} ERROR: keep alive lru size: 509 max: 500 total: 571 [Nov 8 22:46:05.797] Server {0x2b8808b0a700} ERROR: keep alive lru size: 505 max: 500 total: 561 [Nov 8 22:46:05.889] Server {0x2b8808403700} ERROR: keep alive lru size: 509 max: 500 total: 552 [Nov 8 22:46:05.889] Server {0x2b8803847700} ERROR: keep alive lru size: 504 max: 500 total: 560 [Nov 8 22:46:05.889] Server {0x2b8803b4a700} ERROR: keep alive lru size: 515 max: 500 total: 562 [Nov 8 22:46:05.889] Server {0x2b8808605700} ERROR: keep alive lru size: 507 max: 500 total: 564 [Nov 8 22:46:05.889] Server {0x2b8803948700} ERROR: keep alive lru size: 510 max: 500 total: 572
After the discussions at IETF, I really feel we should generalize this, and augment a lot of the inactivity stuff with LRUs. We would have three LRUs:
1. KA Idle connection LRU
2. Inactive Connection LRU
3. Active Connection LRU
Instead of creating / canceling events for these, we'd simply move connections between these LRUs as things changes. When you hit the max number of (configured) client connections, you start in the 1. LRU, and try to reap one there. If there's nothing to reap, you go to the next LRU (2.) and try to reap one there. A question would be if it ever makes sense to reap an Active Connection (3.), but that's something that could be configurable.
In addition to this, we could still honor maximum timeouts for each of these LRUs. Since each LRU is sorted (by definition), it's easy to process the bottom portion of each LRU and reap those which are "older" that the configured maximum. This can be done fairly infrequently (say every 5-10s at the most, configurable), and would be efficient since you only have to consider those entries older than a certain age.
This combined I think can eliminate most (all?) of inactivity cop. It also should (I hope?) be cheaper to maintain these LRUs than the creation and canceling of all the thousands (millions?) of events constantly. Since a NetVC can only be in one LRU at any time, it's trivial to manage this, the patch already adds the plumbing for managing this. What we need is just a state variable to indicate which LRU the VC is in.
I'd really like to hear from John Plevyak on this bug, and see what he thinks about moving these inactivity cop / events processing into the general concept of LRU. Other than performance, the biggest benefit with this work is that it would be a huge improvement for how people operate and manage their ATS installations and configurations (no more guessing on timeouts vs conn throttle setting).
Thanks Bryan for providing this patch!
I spoke with John about this before, and he seemed favorable to these ideas. I really think we should generalize this, such that we move VCs between LRUs instead of inactivity / timeout events.
From a configuration perspective, what the user generally keeps about is to not run out of file descriptors; In a sense, you configure your system based on a fixed set of resources, and trying to guess timeout values is virtually impossible.
There's still room for timeouts, such that you also enforce certain max lifetimes in each of the LRUs. But this would be easily to clean out, walking the LRUs periodically (once a second?) starting from the bottom. Since the LRUs are by definition ordered, this is very efficient; You only traverse entries that would be evicted, and nothing else.
I know I'm repeating myself here, but I'm hugely positive to these changes, and I think this can enable us to handle a large number of VCs with little overhead and significantly easier management.
Commit ce08ae8a88b15cda917484197e75e524456d0a5d in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=ce08ae8 ]
TS-153: "Dynamic" keep-alive timeouts for incoming connections
I think remove_from_keep_alive_lru() should also be called from HttpClientSession::do_io_close(), else there may be a danger of a double release for an already released netvc?
Sorry, please ignore the comment - I see that remove_from_keep_alive_lru() is being called from {{UnixNetVConnection::do_io_close}.
Commit 04871a9742ccb055ddfbd7f0dbe1cce4c3348cb6 in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=04871a9 ]
TS-153: fixed singed/unsinged compare
I have updates after running this feature in production for ~1 week.
Changes:
1. Changed the limit on the number of connections to include all incoming connections instead of just ones that are idle. This will keep the number of incoming connections per thread more consistent.
2. Change the double link list to a queue.
3. When adding to the queue if the connection is already in the queue remove it and then add it to the end the queue.
4. Properly close the connection by mimicking what inactivity cop does to close the connection.
5. Add stats to determine the average KA timeout since it is now dynamic.
6. Added support for spdy connections.
New config name:
proxy.config.net.max_connections_in
New stats:
proxy.process.net.dynamic_keep_alive_timeout_in_total
proxy.process.net.dynamic_keep_alive_timeout_in_count
amc/sudheerv came up with proxy.config.connections.threshold_shed_idle - I like that, its succinct and is very descriptive about whats accomplished with this setting
proxy.config.connections doesn't exit that is why I picked proxy.config.net. Also, the name should have _in in it since it is for incoming connections:
I would be find with proxy.config.net.threshold_shed_idle_in
Thanks, bcall - Yes, I think proxy.config.net.threshold_shed_idle_in sounds better, although, it doesn't say what it's shedding - would proxy.config.net.connections.threshold_shed_idle_in sound better?
Commit 5e91acd24e67da73baa81ded14878e32273f1e20 in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=5e91acd ]
TS-153: Dynamic keep-alive timeouts
1. Changed the limit on the number of connections to include all
incoming connections instead of just ones that are keep-alive. This will
keep the number of incoming connections per thread more consistent.
However, it will only close keep-alive connections.
2. Change the double link list to a queue.
3. When adding to the queue if the connection is already in the queue
remove it and then add it to the end the queue.
4. Properly close the connection by mimicking what inactivity cop does
to close the connection.
5. Add stats to determine the average KA timeout since it is now dynamic.
6. Added support for spdy connections.
Config option is now:
proxy.config.net.connections.threshold_shed_idle_in
Stats added are:
proxy.process.net.dynamic_keep_alive_timeout_in_total
proxy.process.net.dynamic_keep_alive_timeout_in_count
From the conversations on IRC, I really feel we need a better name. I like bcall's proposal of
max_connections_in max_connections_out max_connections_active_in max_connections_active_out
When we started the discussions around these features, the vision what that we'd remove the current "throttle" mechanism, and instead the number of FDs we require is the sum of the above. And, we throttle accordingly on the sums of "_in" and "_out".
The point here is to simplify both configurations, and code. What I had envisioned was that we allow up to the "sum" of the _in client connections. Once we hit that, we start shedding; The priority on shedding is
1. Shed the oldest KA connections
2. Shed the oldest inactive connection
3. Shed the oldest active connection (this must be optional)
If after this there is still no free socket / FD, we throttle (deny the connection).
This would allow us to get rid of a significant amount of timeout shenanigans in the event system. There's no point (IMO) to ever timeout on KA connections, they will get shedded when needed. The other two types of connections would still have timeouts of course, so those configs can remain. But, we can perhaps change the code to be less aggressive on how often we update / cancel such timeouts (i.e. we probably don't have to do it on higher resolution than seconds).
Commit 1dfc029c78bd986e42fcd10a2a30f94aa8f54e7a in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=1dfc029 ]
TS-153: Updates to traffic_top to report dynamic KA times
Commit 0b7bf112a778111469f837a3fbba3982d217bb5d in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=0b7bf11 ]
TS-153: Dynamic keep-alive timeouts
Renaming proxy.config.net.connections.threshold_shed_idle_in to
proxy.config.net.max_connections_in to be inline with the overall
design in TS-3312
Commit ab3fec0a88aba5bf8da5078ea0ab326e4fd61988 in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=ab3fec0 ]
TS-153: Dynamic keep-alive timeouts
Updates after a code review with AMC and more testing
Commit 9763a7f87a1cf7e3a6df20edc7bf19bf99965663 in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=9763a7f ]
TS-153: Dynamic keep-alive timeouts
Removed an unneed removal from the LRU.
Commit b48c343e6bbc58527b875fef31cbc1c8349fc650 in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=b48c343 ]
TS-153: Forgot to add the abort after removing the assert when a stat
can't be fetched in traffic_top
Commit abe6035641ef4b00e14c3e9b87fc616eba345a03 in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=abe6035 ]
TS-153: Fixed warning of unused argument
Commit c53eb2e011b271db672c299b8fa4049be210ce4b in trafficserver's branch refs/heads/master from bcall
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=c53eb2e ]
TS-153: Fixed typo on method name
Added LRU for Keep-Alive and the ability to limit the number of overall connections by only closing Keep-Alive connections for incoming (client) connections. I will create another ticket to extend the functionality to limit the number of active connections and add the ability to do this for the outgoing connections.
Moving to 3.3.2.