[TS-1487] the ordering of plugin_init and init_HttpProxyServer cause crashed TS to core endlessly

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.5
• Component/s: Core
• Labels: A

Environment

Linux RHEL6.2

• Assignee: Alan M. Carroll
• Reporter: Aidan McGurn
• Created: 20/Sep/12 19:18
• Updated: 09/Aug/13 14:34
• Resolved: 18/Jul/13 22:46

Description

We've had a serious issue whereby the TS when it crashes re-spawns/cores continuously when its tries to re-start under load. I traced the issue to SNMP research library (a third party lib)- They use selects and what happens is the file descriptor number spikes under load after the crash as all the sockets get opened at once - this causes buffer overflow in the select (which their library is full of) as the fd allocated to the FD_SET is much bigger than the FD_SETSIZE of 1024 (which was a bitch to track down as the stack was corrupted and gdb therefore useless). Tracing why this happened on 3.2.0 and not 3.0.2, I find the sequence
of the plugin_init has changed - On 3.0.2 the sequence was in effect 1. plugin_init and then 2. init_HttpProxyServer. Whereas this has mysteriously been reversed on 3.2.0. In order to get our system to work in this crash case , I've patched ATS to flip them around like in 3.0.2.
i'll attach the patch we propose we need to use to get around this.

Is this actually a bug then waiting to happen in other systems - Or was there a reason to change this sequence?

Issue Links

• contains: TS-2035 traffic server cores on startup when enabling multiple ports