Major Description
Hi,
when ICP is enabled and ATS is under huge load a "service trafficserver stop" followed by a "service trafficserver start" cause a crashing loop (ATS crash, restart, crash again, restart, crash again... )
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local' traffic_server: using root directory '/usr/local' traffic_server: Segmentation fault (Address not mapped to object [(nil)])traffic_server - STACK TRACE: traffic_server: Segmentation fault (Address not mapped to object [(nil)])traffic_server - STACK TRACE: /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809] /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809] /lib64/libc.so.6[0x31866326a0] /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16] /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe] /lib64/libc.so.6[0x31866326a0] /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16] /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xbde)[0x59cd4e] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] /usr/local/bin/traffic_server(_ZN6HttpSM21state_cache_open_readEiPv+0x15c)[0x59a3fc] /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8] /usr/local/bin/traffic_server(_ZN11HttpCacheSM21state_cache_open_readEiPv+0x152)[0x5760a2] /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePKci+0x2b1)[0x6e26c1] /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0xbd)[0x57574d] /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x142)[0x5831d2] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7b6)[0x59c926] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x229)[0x59c399] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x3dc)[0x59371c] /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8] /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2b2)[0x595b22] /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x75f)[0x59d8cf] /usr/local/bin/traffic_server(_ZN17HttpClientSession15new_transactionEv+0xec)[0x5769dc] /usr/local/bin/traffic_server(_ZN18ProxyClientSession14do_api_calloutE12TSHttpHookID+0x14d)[0x4e585d] /usr/local/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionP9MIOBufferP14IOBufferReaderb+0x224)[0x578654] /usr/local/bin/traffic_server(_ZN17HttpSessionAccept6acceptEP14NetVConnectionP9MIOBufferP14IOBufferReader+0x1ea)[0x5729ea] /usr/local/bin/traffic_server(_ZN23ProtocolProbeTrampoline17ioCompletionEventEiPv+0x483)[0x4e5343] /usr/local/bin/traffic_server[0x71eaf9] /usr/local/bin/traffic_server[0x723c0e] /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x7151d2] /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x125)[0x742d05] /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4c9)[0x743649] /usr/local/bin/traffic_server[0x74217a] /lib64/libpthread.so.0(+0x3186a079d1)[0x2b864a19f9d1] /lib64/libc.so.6(clone+0x6d)[0x31866e88fd] /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e] /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a]
Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: --- Manager Starting --- Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27) Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: --- traffic_server Starting --- Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47) Sep 23 16:12:07 redis-01 kernel: [ET_NET 4][22681]: segfault at 0 ip 00000000004aca16 sp 00002b3bad5ceb00 error 4 in traffic_server[400000+415000] Sep 23 16:12:10 redis-01 abrt[22993]: Saved core dump of pid 22676 (/usr/local/bin/traffic_server) to /var/spool/abrt/ccpp-2015-09-23-16:12:07-22676 (913047552 bytes) Sep 23 16:12:10 redis-01 abrtd: Directory 'ccpp-2015-09-23-16:12:07-22676' creation detected Sep 23 16:12:10 redis-01 abrtd: Executable '/usr/local/bin/traffic_server' doesn't belong to any package and ProcessUnpackaged is set to 'no' Sep 23 16:12:10 redis-01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676' exited with 1 Sep 23 16:12:10 redis-01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676' Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe) Sep 23 16:12:10 redis-01 traffic_cop[22665]: cop received child status signal [22667 256] Sep 23 16:12:10 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead Sep 23 16:12:10 redis-01 traffic_cop[22665]: spawning traffic_manager Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: --- Manager Starting --- Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27) Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: --- traffic_server Starting --- Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47) Sep 23 16:12:15 redis-01 kernel: [ET_NET 255][23280]: segfault at 0 ip 00000000004aca16 sp 00002b7d2aee6b00 error 4 in traffic_server[400000+415000] Sep 23 16:12:15 redis-01 abrt[23339]: Not saving repeating crash in '/usr/local/bin/traffic_server' Sep 23 16:12:17 redis-01 abrt[23339]: Saved core dump of pid 23024 to core.23024 (906752000 bytes) Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe) Sep 23 16:12:17 redis-01 traffic_cop[22665]: cop received child status signal [23012 256] Sep 23 16:12:17 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead Sep 23 16:12:17 redis-01 traffic_cop[22665]: spawning traffic_manager Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: --- Manager Starting --- Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27) Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: --- traffic_server Starting --- Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47) Sep 23 16:12:22 redis-01 kernel: [ET_NET 193][23562]: segfault at 0 ip 00000000004aca16 sp 00002b866aaa2b00 error 4 in traffic_server[400000+415000] Sep 23 16:12:22 redis-01 abrt[23684]: Not saving repeating crash in '/usr/local/bin/traffic_server' Sep 23 16:12:24 redis-01 abrt[23684]: Saved core dump of pid 23368 to core.23368 (910950400 bytes) Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104) Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe) Sep 23 16:12:25 redis-01 traffic_cop[22665]: cop received child status signal [23356 256] Sep 23 16:12:25 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead Sep 23 16:12:25 redis-01 traffic_cop[22665]: unable to start traffic_manager, retrying in 60 second(s)
Disabling ICP solve the issue.
I have never used and tested ICP before (older releases).