[TS-4817] Frequent segfaults in Cache::open_write

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 7.1.0
• Component/s: Cache, HTTP
• Assignee: Unassigned
• Reporter: Mathias Biilmann Christensen
• Created: 06/Sep/16 10:00
• Updated: 28/Sep/16 21:36

Description

I've been running some tests with the master branch sending some production traffic to a test server, and am seeing very frequent segfaults.

I'm running fb8bcbcac3c1d6c7a15340f0093342fb9f207e78

The stack traces look like this:

traffic_server - STACK TRACE: 
/opt/ts/bin/traffic_server(crash_logger_invoke(int, siginfo_t*, void*)+0xc3)[0x511280]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x2ae12de39330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x2ae12eaa1c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x2ae12eaa5028]
/opt/ts/lib/libtsutil.so.7(ink_warning(char const*, ...)+0x0)[0x2ae12cb11f4e]
/opt/ts/lib/libtsutil.so.7(ats_base64_encode(unsigned char const*, unsigned long, char*, unsigned long, unsigned long*)+0x0)[0x2ae12cb0f8e4]
/opt/ts/bin/traffic_server(HttpTunnel::consumer_handler(int, HttpTunnelConsumer*)+0xcc)[0x63efb2]
/opt/ts/bin/traffic_server(HttpTunnel::main_handler(int, void*)+0x133)[0x63f9b7]
/opt/ts/bin/traffic_server(Continuation::handleEvent(int, void*)+0x72)[0x51425a]
/opt/ts/bin/traffic_server(CacheVC::calluser(int)+0xaa)[0x74f4c8]
/opt/ts/bin/traffic_server(CacheVC::openWriteMain(int, Event*)+0x3de)[0x759d48]
/opt/ts/bin/traffic_server(Continuation::handleEvent(int, void*)+0x72)[0x51425a]
/opt/ts/bin/traffic_server(CacheVC::callcont(int)+0xfc)[0x74f608]
/opt/ts/bin/traffic_server(CacheVC::openWriteStartDone(int, Event*)+0x8a4)[0x75a9ee]
/opt/ts/bin/traffic_server(Continuation::handleEvent(int, void*)+0x72)[0x51425a]
/opt/ts/bin/traffic_server(CacheVC::handleReadDone(int, Event*)+0xd15)[0x72a3e7]
/opt/ts/bin/traffic_server(Continuation::handleEvent(int, void*)+0x72)[0x51425a]
/opt/ts/bin/traffic_server(Cache::open_write(Continuation*, ats::CryptoHash const*, HTTPInfo*, long, ats::CryptoHash const*, CacheFragType, char const*, int)+0x6f$
)[0x75b9df]
/opt/ts/bin/traffic_server(CacheProcessor::open_write(Continuation*, int, HttpCacheKey const*, bool, HTTPHdr*, HTTPInfo*, long, CacheFragType)+0x129)[0x72eee5]
/opt/ts/bin/traffic_server(HttpCacheSM::open_write(HttpCacheKey const*, URL*, HTTPHdr*, HTTPInfo*, long, bool, bool)+0x236)[0x5d25a2]
/opt/ts/bin/traffic_server(HttpSM::do_cache_prepare_action(HttpCacheSM*, HTTPInfo*, bool, bool)+0x352)[0x5f054e]
/opt/ts/bin/traffic_server(HttpSM::do_cache_prepare_write_transform()+0x69)[0x6013dd]
/opt/ts/bin/traffic_server(HttpSM::set_next_state()+0x1a27)[0x5fb613]
/opt/ts/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*))+0x1ae)[0x5f9be4]
/opt/ts/bin/traffic_server(HttpSM::state_response_wait_for_transform_read(int, void*)+0x19b)[0x5e35cd]
/opt/ts/bin/traffic_server(HttpSM::main_handler(int, void*)+0x33a)[0x5e88ac]
/opt/ts/bin/traffic_server(Continuation::handleEvent(int, void*)+0x72)[0x51425a]
/opt/ts/bin/traffic_server(TransformTerminus::handle_event(int, void*)+0x2fe)[0x561096]
/opt/ts/bin/traffic_server(Continuation::handleEvent(int, void*)+0x72)[0x51425a]
/opt/ts/bin/traffic_server(EThread::process_event(Event*, int)+0x136)[0x7b3eac]
/opt/ts/bin/traffic_server(EThread::execute()+0xdc)[0x7b410c]
/opt/ts/bin/traffic_server(main+0x139c)[0x546c99]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x2ae12ea8cf45]
/opt/ts/bin/traffic_server[0x4f9840]
FATAL: HttpTunnel.cc:1372: failed assertion `c->alive == true`

I'm attaching a core dump from a debug build of ATS as well

Issue Links

• is related to: TS-4813 HttpTunnel.cc:1215: failed assertion `p->alive == true || event == HTTP_TUNNEL_EVENT_PRECOMPLETE ...