[TC-197] File descriptor leak caused by NIO protocol connector

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0, 2.2.0
• Fix Version/s: 2.1.0, 2.2.0
• Component/s: Traffic Router
• Labels: file_descriptor_leak
• Assignee: Unassigned
• Reporter: Jeff Elsloo
• Created: 15/Mar/17 20:12
• Updated: 16/Aug/17 15:14

Description

The new default configuration for Traffic Router and Tomcat is to use the Http11NioProtocol adapter over the previous Http11Protocol adapter. This is accomplished via the LanguidNioProtocol connectors specified in Tomcat's server.xml.

We observed a file descriptor leak with connections, but have not dug into the cause. We observed roughly 30-40k connections in CLOSE_WAIT which caused the machine to exhaust its file descriptors based upon configured limits. We also observed a similar growth curve on total threads, so something is not behaving and being cleaned up appropriately.

The cause could range from a bug in the Http11NioProtocol connector due to the ancient version of Tomcat we're using, tuning within the connector itself (max threads, intervals, etc), tuning within the JVM (memory, GC, etc), or at the OS level (kernel params around TCP connections, etc).