Description
When testing master on production hosts, noticed the below crash occuring repeatedly every time ats version is changed. This crash stops happening after clearing the cache. This needs further investigation, but, I remember a discussion between briang and zwoop about duplicate string fields in HdrHeap. Not sure if this core is related to that. Would appreciate if briang or zwoop can comment. Thank you.
[example_prep.sh] Checking/Moving old cores...
[TrafficServer] using root directory '/home/y'
NOTE: Traffic Server received Sig 11: Segmentation fault
/home/y/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0x30d3c0f500)[0x2aae27e2d500]
/home/y/bin/traffic_server(_ZN7HdrHeap20inherit_string_heapsEPKS_+0x271)[0x61caa1]
/home/y/bin/traffic_server(_Z14http_hdr_cloneP11HTTPHdrImplP7HdrHeapS2_+0x93)[0x619f83]
/home/y/bin/traffic_server(_ZN19HttpTransactHeaders18copy_header_fieldsEP7HTTPHdrS1_bl+0x1be)[0x5c201e]
/home/y/bin/traffic_server(_ZN12HttpTransact14build_responseEPNS_5StateEP7HTTPHdrS3_11HTTPVersion10HTTPStatusPKc+0x3ed)[0x5a287d]
/home/y/bin/traffic_server(_ZN12HttpTransact25build_response_from_cacheEPNS_5StateE15HTTPWarningCode+0x354)[0x5b67f4]
/home/y/bin/traffic_server(_ZN12HttpTransact22HandleCacheOpenReadHitEPNS_5StateE+0x448)[0x5b84c8]
/home/y/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x66)[0x573816]
/home/y/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2d2)[0x58ba72]
/home/y/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x2b0)[0x583070]
/home/y/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x1ea)[0x58d06a]
/home/y/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2d2)[0x58ba72]
/home/y/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x2b0)[0x583070]
/home/y/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x1ea)[0x58d06a]
/home/y/bin/traffic_server(_ZN6HttpSM21state_cache_open_readEiPv+0xfe)[0x58533e]
/home/y/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x587de8]
/home/y/bin/traffic_server(_ZN11HttpCacheSM21state_cache_open_readEiPv+0x1b2)[0x566e12]
/home/y/bin/traffic_server(_ZN7CacheVC8callcontEi+0x53)[0x653453]
/home/y/bin/traffic_server(_ZN7CacheVC17openReadStartHeadEiP5Event+0x7cf)[0x6be9af]
/home/y/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x1ed)[0x69d40d]
/home/y/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x35)[0x6539c5]
/home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x714aef]
/home/y/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x71561b]
/home/y/bin/traffic_server[0x713e9a]
/lib64/libpthread.so.0(+0x30d3c07851)[0x2aae27e25851]
/lib64/libc.so.6(clone+0x6d)[0x30d38e890d]
gdb output below:
(gdb) bt #0 ink_atomic_increment<int, int> (this=0x2afb60113010, inherit_from=0x2afaa824e688) at ../../lib/ts/ink_atomic.h:162 #1 refcount_inc (this=0x2afb60113010, inherit_from=0x2afaa824e688) at ../../lib/ts/Ptr.h:279 #2 operator= (this=0x2afb60113010, inherit_from=0x2afaa824e688) at ../../lib/ts/Ptr.h:408 #3 attach_str_heap (this=0x2afb60113010, inherit_from=0x2afaa824e688) at HdrHeap.cc:1000 #4 HdrHeap::inherit_string_heaps (this=0x2afb60113010, inherit_from=0x2afaa824e688) at HdrHeap.cc:1081 #5 0x0000000000619f83 in http_hdr_clone (s_hh=0x2afaa824e710, s_heap=0x2afaa824e688, d_heap=0x2afb60113010) at HTTP.cc:375 #6 0x00000000005c201e in copy (src_hdr=0x2afaa824e0b8, new_hdr=0x2afac4058c50, retain_proxy_auth_hdrs=false, date=0) at ../../proxy/hdrs/HTTP.h:867 #7 HttpTransactHeaders::copy_header_fields (src_hdr=0x2afaa824e0b8, new_hdr=0x2afac4058c50, retain_proxy_auth_hdrs=false, date=0) at HttpTransactHeaders.cc:201 #8 0x00000000005a287d in HttpTransact::build_response (s=0x2afac4058570, base_response=0x2afaa824e0b8, outgoing_response=0x2afac4058c50, outgoing_version=<value optimized out>, status_code=HTTP_STATUS_NONE, reason_phrase=0x7323ac "None") at HttpTransact.cc:7926 #9 0x00000000005b67f4 in HttpTransact::build_response_from_cache (s=0x2afac4058570, warning_code=HTTP_WARNING_CODE_NONE) at HttpTransact.cc:2925 #10 0x00000000005b84c8 in HttpTransact::HandleCacheOpenReadHit (s=0x2afac4058570) at HttpTransact.cc:2811 #11 0x0000000000573816 in HttpSM::call_transact_and_set_next_state (this=0x2afac4058500, f=<value optimized out>) at HttpSM.cc:6748 #12 0x000000000058ba72 in HttpSM::handle_api_return (this=0x2afac4058500) at HttpSM.cc:1501 #13 0x0000000000583070 in HttpSM::state_api_callout (this=0x2afac4058500, event=60000, data=0x0) at HttpSM.cc:1433 #14 0x0000000000587fdb in HttpSM::state_api_callback (this=0x2afac4058500, event=60000, data=0x0) at HttpSM.cc:1251 #15 0x00000000004c0acc in TSHttpTxnReenable (txnp=0x2afac4058500, event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5548 #16 0x00002afa68cb19b7 in yca_cache_access_control (contp=<value optimized out>, event=<value optimized out>, edata=<value optimized out>) at yca_plugin.cpp:747 #17 yca_auth_plugin_local_handler (contp=<value optimized out>, event=<value optimized out>, edata=<value optimized out>) at yca_plugin.cpp:955 #18 0x0000000000582ed4 in HttpSM::state_api_callout (this=0x2afac4058500, event=<value optimized out>, data=<value optimized out>) at HttpSM.cc:1358 #19 0x000000000058d06a in HttpSM::set_next_state (this=0x2afac4058500) at HttpSM.cc:6790 #20 0x000000000058ba72 in HttpSM::handle_api_return (this=0x2afac4058500) at HttpSM.cc:1501 #21 0x0000000000583070 in HttpSM::state_api_callout (this=0x2afac4058500, event=0, data=0x0) at HttpSM.cc:1433 #22 0x000000000058d06a in HttpSM::set_next_state (this=0x2afac4058500) at HttpSM.cc:6790 #23 0x000000000058533e in HttpSM::state_cache_open_read (this=0x2afac4058500, event=1102, data=0x2afbd4006e80) at HttpSM.cc:2423 #24 0x0000000000587de8 in HttpSM::main_handler (this=0x2afac4058500, event=1102, data=0x2afbd4006e80) at HttpSM.cc:2482 #25 0x0000000000566e12 in handleEvent (this=0x2afac4059ed8, event=<value optimized out>, data=0x2afbd4006e80) at ../../iocore/eventsystem/I_Continuation.h:146 #26 HttpCacheSM::state_cache_open_read (this=0x2afac4059ed8, event=<value optimized out>, data=0x2afbd4006e80) at HttpCacheSM.cc:118 #27 0x0000000000653453 in handleEvent (this=0x2afbd4006e80, event=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:146 #28 CacheVC::callcont (this=0x2afbd4006e80, event=<value optimized out>) at ../../iocore/cache/P_CacheInternal.h:657 #29 0x00000000006be9af in CacheVC::openReadStartHead (this=0x2afbd4006e80, event=3900, e=0x0) at CacheRead.cc:1206 #30 0x000000000069d40d in handleEvent (this=0x2afbd4006e80, event=<value optimized out>, e=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:146 #31 CacheVC::handleReadDone (this=0x2afbd4006e80, event=<value optimized out>, e=<value optimized out>) at Cache.cc:2521 #32 0x00000000006539c5 in handleEvent (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:146 #33 AIOCallbackInternal::io_complete (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at ../../iocore/aio/P_AIO.h:123 #34 0x0000000000714aef in handleEvent (this=0x2afa54b18010, e=0x2afaac01ac90, calling_code=1) at I_Continuation.h:146 #35 EThread::process_event (this=0x2afa54b18010, e=0x2afaac01ac90, calling_code=1) at UnixEThread.cc:145 #36 0x000000000071561b in EThread::execute (this=0x2afa54b18010) at UnixEThread.cc:196 #37 0x0000000000713e9a in spawn_thread_internal (a=0x16c3220) at Thread.cc:88 #38 0x00002afa4f271851 in start_thread () from /lib64/libpthread.so.0 #39 0x000000361a0e894d in clone () from /lib64/libc.so.6 (gdb) print mem $1 = (volatile int *) 0x70d87d9700000008 (gdb) print *mem Cannot access memory at address 0x70d87d9700000008 (gdb) up #1 refcount_inc (this=0x2afb60113010, inherit_from=0x2afaa824e688) at ../../lib/ts/Ptr.h:279 279 ../../lib/ts/Ptr.h: No such file or directory. in ../../lib/ts/Ptr.h (gdb) print this $2 = (RefCountObj * const) 0x70d87d9700000000 (gdb) print *this Cannot access memory at address 0x70d87d9700000000 (gdb) up #2 operator= (this=0x2afb60113010, inherit_from=0x2afaa824e688) at ../../lib/ts/Ptr.h:408 408 in ../../lib/ts/Ptr.h (gdb) print this $3 = (Ptr<RefCountObj> * const) 0x2afb60113060 (gdb) print *this $4 = {m_ptr = 0x70d87d9700000000} (gdb) up #3 attach_str_heap (this=0x2afb60113010, inherit_from=0x2afaa824e688) at HdrHeap.cc:1000 1000 HdrHeap.cc: No such file or directory. in HdrHeap.cc (gdb) print this $5 = (HdrHeap * const) 0x2afb60113010 (gdb) print *this $6 = {m_magic = 2882404077, m_free_start = 0x2afb60113360 "\002p", m_data_start = 0x2afb601130e0 "\003\060", m_size = 2048, m_writeable = true, m_next = 0x0, m_free_size = 1200, m_read_write_heap = {m_ptr = 0x0}, m_ronly_heap = {{m_ref_count_ptr = {m_ptr = 0x2afa6c0019a0}, m_heap_start = 0x2afaa824e990 "HTTP/1.1 200 OK\r\nDate: Mon, 05 May 2014 19:02:05 GMT\r\nP3P: policyref=\"http://info.yahoo.com/w3c/p3p.xml\", CP=\"CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi "..., m_heap_len = 539, m_locked = false}, {m_ref_count_ptr = {m_ptr = 0x70d87d9700000000}, m_heap_start = 0x0, m_heap_len = 0, m_locked = false}, {m_ref_count_ptr = {m_ptr = 0x0}, m_heap_start = 0x0, m_heap_len = 0, m_locked = false}, {m_ref_count_ptr = {m_ptr = 0x0}, m_heap_start = 0x0, m_heap_len = 0, m_locked = false}, {m_ref_count_ptr = {m_ptr = 0x0}, m_heap_start = 0x0, m_heap_len = 0, m_locked = false}, {m_ref_count_ptr = {m_ptr = 0x0}, m_heap_start = 0x0, m_heap_len = 0, m_locked = false}}, m_lost_string_space = 0} (gdb)
Attachments
Issue Links
- Is contained by
-
TS-2766 HdrHeap::coalesce_str_heaps doesn't properly calculate new heap size
-
- Closed
-