[TS-1272] workaround for - FATAL: HttpSM.cc:890: failed assert `0`

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.3
• Fix Version/s: 3.1.4
• Component/s: HTTP
• Labels: regression

Environment

Linux hostname.domain.com 2.6.32-220.el6.x86_64 #1 SMP Wed Dec 7 10:41:06 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

• Assignee: James Peach
• Reporter: James Peach
• Created: 25/May/12 04:33
• Updated: 06/Jun/12 00:01
• Resolved: 25/May/12 04:42

Description

We have a 3.1.3 installation running against a light production load and are occasionally getting this failed assertion:

FATAL: HttpSM.cc:890: failed assert `0`
/usr/bin/traffic_server - STACK TRACE:
/usr/lib64/trafficserver/libtsutil.so.3(ink_fatal+0x88)[0x2b3b69c88538]
/usr/lib64/trafficserver/libtsutil.so.3(_ink_assert+0x1f)[0x2b3b69c86c6f]
/usr/bin/traffic_server(_ZN6HttpSM28state_watch_for_client_abortEiPv+0x7d)[0x52ae0d]
/usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8)[0x536008]
/usr/bin/traffic_server(_ZN18UnixNetVConnection14readSignalDoneEiP10NetHandler+0x3d)[0x678e2d]
/usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x6ef)[0x66cabf]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x25a)[0x6737ba]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0xb4)[0x69cc34]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x4c3)[0x69d5c3]
/usr/bin/traffic_server[0x69bb72]
/lib64/libpthread.so.0[0x3170e077f1]
/lib64/libc.so.6(clone+0x6d)[0x3170ae570d]

This causes a corruption of the disk cache every time it occurs, such that whenever the traffic server starts back up, it gets into a crash loop and does this over and over each time it tries to start up:

NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0[0x3170e0f4a0]
/usr/bin/traffic_server(_Z15dir_clear_rangellP3Vol+0x6b)[0x63370b]
/usr/bin/traffic_server(_ZN3Vol24handle_recover_from_dataEiPv+0x531)[0x62c5d1]
/usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x35)[0x62cfb5]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0xb4)[0x69cc34]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x5ab)[0x69d6ab]
/usr/bin/traffic_server[0x69bb72]
/lib64/libpthread.so.0[0x3170e077f1]
/lib64/libc.so.6(clone+0x6d)[0x3170ae570d]
[Apr 29 17:39:57.149] Manager

{0x7f4d3481c7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Apr 29 17:39:57.149] Manager {0x7f4d3481c7e0}

FATAL: (last system error 104: Connection reset by peer)
[Apr 29 17:39:57.149] Manager

{0x7f4d3481c7e0} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
[Apr 29 17:39:57.149] Manager {0x7f4d3481c7e0}

NOTE: [LocalManager::processShutdown] Executing process shutdown request.
[Apr 29 17:39:57.149] Manager

{0x7f4d3481c7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Apr 29 17:39:57.149] Manager {0x7f4d3481c7e0}

ERROR: (last system error 32: Broken pipe)

The configuration for this setup is pretty minimal, and is mostly a single remap rule that chains together the remap_regex and conf_remap plugins to turn off auth for files ending in a particular extension.

I will add any appropriate configs, logs, etc as needed to help troubleshoot this.

Issue Links

• is a clone of: TS-1233 FATAL: HttpSM.cc:890: failed assert `0`
• is broken by: TS-841 support TLS NextProtocol negotiation