[TS-4742] FATAL: ClusterCache.cc:1759: failed assert `tc->tunnel_cont == tc`

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: sometime
• Component/s: Clustering
• Assignee: Unassigned
• Reporter: Paul Stengel
• Created: 11/Aug/16 20:34
• Updated: 18/Aug/16 03:43

Description

We're running TS in a two-server cluster right now. The clustering works well and is functioning as you'd expect. However, every so often, TS crashes with the following backtrace:

FATAL: ClusterCache.cc:1759: failed assert `tc->tunnel_cont == tc`
traffic_server: using root directory '/usr'
traffic_server: Aborted (Signal sent by tkill() 13260 114)
traffic_server - STACK TRACE: 
/usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x55bbaed08f00]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x2b45c2ff13d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x2b45c3cd7418]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x2b45c3cd901a]
/usr/lib/trafficserver/libtsutil.so.6(+0x2430f)[0x2b45c1c8f30f]
/usr/lib/trafficserver/libtsutil.so.6(+0x243a9)[0x2b45c1c8f3a9]
/usr/lib/trafficserver/libtsutil.so.6(+0x22595)[0x2b45c1c8d595]
/usr/bin/traffic_server(+0x2ee7e8)[0x55bbaef207e8]
/usr/bin/traffic_server(_ZN12OneWayTunnel10startEventEiPv+0x16f)[0x55bbaeecd9df]
/usr/bin/traffic_server(_ZN14ClusterHandler32cluster_signal_and_update_lockedEiP18ClusterVConnectionP17ClusterVConnState+0x2f)[0x55bbaef05abf]
/usr/bin/traffic_server(_ZN14ClusterHandler20valid_for_data_writeEP18ClusterVConnection+0x8fa)[0x55bbaeefff1a]
/usr/bin/traffic_server(_ZN14ClusterHandler23build_write_descriptorsEv+0x171)[0x55bbaef00091]
/usr/bin/traffic_server(_ZN14ClusterHandler13process_writeElb+0x578)[0x55bbaef00868]
/usr/bin/traffic_server(_ZN14ClusterHandler16mainClusterEventEiP5Event+0x195)[0x55bbaef00ae5]
/usr/bin/traffic_server(_ZN12ClusterState10IOCompleteEv+0xc7)[0x55bbaef04be7]
/usr/bin/traffic_server(_ZN12ClusterState16doIO_write_eventEiPv+0x119)[0x55bbaef04fd9]
/usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x1191)[0x55bbaefabfd1]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x328)[0x55bbaef98ad8]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x55bbaefdd362]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x679)[0x55bbaefde079]
/usr/bin/traffic_server(+0x3aae49)[0x55bbaefdce49]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa)[0x2b45c2fe76fa]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2b45c3da8b5d]

We've not found a way to reproduce the crash. Here are the diags.log entries that show when the server starts back up (indicating a crash and recovery):

[Aug 11 07:23:46.245] Server {0x2afc9cf458c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 11:14:10.272] Server {0x2ba545a318c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 12:08:51.188] Server {0x2b887a0228c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 14:57:23.900] Server {0x2aad8422b8c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 14:58:17.844] Server {0x2b66aad508c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 15:08:38.752] Server {0x2b09a50cb8c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 15:08:51.001] Server {0x2aee8a7c88c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 15:10:46.796] Server {0x2ace4544e8c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 15:25:27.599] Server {0x2b90b94788c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 15:26:13.642] Server {0x2b814d2088c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 16:03:31.729] Server {0x2b259ff968c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 16:03:44.057] Server {0x2b1cf99198c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 16:15:27.307] Server {0x2b2333b1d8c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 16:39:56.205] Server {0x2b92f018f8c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 16:40:07.535] Server {0x2aad138ab8c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 16:40:17.971] Server {0x2b94640748c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 16:41:07.280] Server {0x2b45c1a798c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 17:01:28.328] Server {0x2adec40e68c0} NOTE: <Main.cc:1902 (main)> traffic server running
[Aug 11 17:06:17.853] Server {0x2aef987758c0} NOTE: <Main.cc:1902 (main)> traffic server running

root@gateway-us-east-1:/var/log/trafficserver$ traffic_server -V
traffic_server: using root directory '/usr'
Apache Traffic Server - traffic_server - 6.2.0 - (build # 080418 on Aug  4 2016 at 18:36:55)

root@gateway-us-east-1:~$ cat /usr/lib/os-release 
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial