Uploaded image for project: 'Traffic Server'
  1. Traffic Server
  2. TS-3943

ATS 6 crash over and over again on restart under load

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 6.0.0
    • Fix Version/s: 7.1.0
    • Component/s: ICP
    • Labels:

      Description

      Hi,

      when ICP is enabled and ATS is under huge load a "service trafficserver stop" followed by a "service trafficserver start" cause a crashing loop (ATS crash, restart, crash again, restart, crash again... )

      [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
      traffic_server: using root directory '/usr/local'
      traffic_server: Segmentation fault (Address not mapped to object [(nil)])traffic_server - STACK TRACE:
      traffic_server: Segmentation fault (Address not mapped to object [(nil)])traffic_server - STACK TRACE:
      /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809]
      /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809]
      /lib64/libc.so.6[0x31866326a0]
      /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16]
      /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe]
      /lib64/libc.so.6[0x31866326a0]
      /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16]
      /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe]
      /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
      /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xbde)[0x59cd4e]
      /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
      /usr/local/bin/traffic_server(_ZN6HttpSM21state_cache_open_readEiPv+0x15c)[0x59a3fc]
      /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8]
      /usr/local/bin/traffic_server(_ZN11HttpCacheSM21state_cache_open_readEiPv+0x152)[0x5760a2]
      /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePKci+0x2b1)[0x6e26c1]
      /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0xbd)[0x57574d]
      /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x142)[0x5831d2]
      /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7b6)[0x59c926]
      /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x229)[0x59c399]
      /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
      /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
      /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x3dc)[0x59371c]
      /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8]
      /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2b2)[0x595b22]
      /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x75f)[0x59d8cf]
      /usr/local/bin/traffic_server(_ZN17HttpClientSession15new_transactionEv+0xec)[0x5769dc]
      /usr/local/bin/traffic_server(_ZN18ProxyClientSession14do_api_calloutE12TSHttpHookID+0x14d)[0x4e585d]
      /usr/local/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionP9MIOBufferP14IOBufferReaderb+0x224)[0x578654]
      /usr/local/bin/traffic_server(_ZN17HttpSessionAccept6acceptEP14NetVConnectionP9MIOBufferP14IOBufferReader+0x1ea)[0x5729ea]
      /usr/local/bin/traffic_server(_ZN23ProtocolProbeTrampoline17ioCompletionEventEiPv+0x483)[0x4e5343]
      /usr/local/bin/traffic_server[0x71eaf9]
      /usr/local/bin/traffic_server[0x723c0e]
      /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x7151d2]
      /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x125)[0x742d05]
      /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4c9)[0x743649]
      /usr/local/bin/traffic_server[0x74217a]
      /lib64/libpthread.so.0(+0x3186a079d1)[0x2b864a19f9d1]
      /lib64/libc.so.6(clone+0x6d)[0x31866e88fd]
      /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
      /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a]
      
      Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: --- Manager Starting ---
      Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27)
      Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: --- traffic_server Starting ---
      Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47)
      Sep 23 16:12:07 redis-01 kernel: [ET_NET 4][22681]: segfault at 0 ip 00000000004aca16 sp 00002b3bad5ceb00 error 4 in traffic_server[400000+415000]
      Sep 23 16:12:10 redis-01 abrt[22993]: Saved core dump of pid 22676 (/usr/local/bin/traffic_server) to /var/spool/abrt/ccpp-2015-09-23-16:12:07-22676 (913047552 bytes)
      Sep 23 16:12:10 redis-01 abrtd: Directory 'ccpp-2015-09-23-16:12:07-22676' creation detected
      Sep 23 16:12:10 redis-01 abrtd: Executable '/usr/local/bin/traffic_server' doesn't belong to any package and ProcessUnpackaged is set to 'no'
      Sep 23 16:12:10 redis-01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676' exited with 1
      Sep 23 16:12:10 redis-01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676'
      Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
      Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
      Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)>  (last system error 32: Broken pipe)
      Sep 23 16:12:10 redis-01 traffic_cop[22665]: cop received child status signal [22667 256]
      Sep 23 16:12:10 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead
      Sep 23 16:12:10 redis-01 traffic_cop[22665]: spawning traffic_manager
      Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: --- Manager Starting ---
      Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27)
      Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: --- traffic_server Starting ---
      Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47)
      Sep 23 16:12:15 redis-01 kernel: [ET_NET 255][23280]: segfault at 0 ip 00000000004aca16 sp 00002b7d2aee6b00 error 4 in traffic_server[400000+415000]
      Sep 23 16:12:15 redis-01 abrt[23339]: Not saving repeating crash in '/usr/local/bin/traffic_server'
      Sep 23 16:12:17 redis-01 abrt[23339]: Saved core dump of pid 23024 to core.23024 (906752000 bytes)
      Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
      Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
      Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)>  (last system error 32: Broken pipe)
      Sep 23 16:12:17 redis-01 traffic_cop[22665]: cop received child status signal [23012 256]
      Sep 23 16:12:17 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead
      Sep 23 16:12:17 redis-01 traffic_cop[22665]: spawning traffic_manager
      Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: --- Manager Starting ---
      Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27)
      Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: --- traffic_server Starting ---
      Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47)
      Sep 23 16:12:22 redis-01 kernel: [ET_NET 193][23562]: segfault at 0 ip 00000000004aca16 sp 00002b866aaa2b00 error 4 in traffic_server[400000+415000]
      Sep 23 16:12:22 redis-01 abrt[23684]: Not saving repeating crash in '/usr/local/bin/traffic_server'
      Sep 23 16:12:24 redis-01 abrt[23684]: Saved core dump of pid 23368 to core.23368 (910950400 bytes)
      Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
      Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
      Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)>  (last system error 32: Broken pipe)
      Sep 23 16:12:25 redis-01 traffic_cop[22665]: cop received child status signal [23356 256]
      Sep 23 16:12:25 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead
      Sep 23 16:12:25 redis-01 traffic_cop[22665]: unable to start traffic_manager, retrying in 60 second(s)
      

      Disabling ICP solve the issue.

      I have never used and tested ICP before (older releases).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              luca.rea Luca Rea
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: