[TS-489] Seg Fault with Connection_Collapsing and clustering enabled.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.6

Environment

Debian Lenny.
2.6.26-2-amd-64
Sun Blade X6240 (2 x Six-Core AMD Opteron(tm) Processor 2439 SE)
64G Memory

• Assignee: mohan_zl
• Reporter: Ricky Chan
• Created: 14/Oct/10 08:07
• Updated: 22/Jul/13 15:28
• Resolved: 09/Feb/11 17:10

Description

Bug is easily reproduced, with the following setup.

Traffic Server 2.0.0
Enable Clustering (so you'll need two machine and make sure cluster is actually working) (LOCAL proxy.local.cluster.type INT 1)
Enable Connection Collapsing (CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 1)

Other changes to records.config which may or may affect it are changes to heuristics:

CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 5
CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400
CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.000100
CONFIG proxy.config.http.cache.fuzz.time INT 240
CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.000005

Using a 3rd machine using apache benchmark (ab) and request with say -n 1000000 with keep alive (-k) and -c 8000 say. I found it happens all the time above 8000. I just fetched a file from origin on lighttpd which had a cache-control header of max-age 86400, so to reduce hitting origin. Size of file is 9 bytes only.

Note: You need to set ulimit -n very high and set sysctl ip_local_port_range to larger than defaults to be able to run test, I did ulimit -n 1000000 and had sysctl -w net.ipv4.ip_local_port_range="1024 65000" to be able to run AB.

Disabling clustering or connection Collapsing the program no longer.

I then added GDB wrapper around traffic_server and it clearly shows it's the connection collapsing API which is at fault here.

I'll add these traces as attachments.