[TS-4832] HTTP/2 delete on stack crash

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: HTTP/2
• Assignee: Unassigned
• Reporter: Susan Hinrichs
• Created: 08/Sep/16 20:27
• Updated: 08/Sep/16 20:30
• Resolved: 08/Sep/16 20:30

Description

A crash we had seen in production.

#0  0x000000000051589a in Mutex_lock (m=0x0, t=0x2b859fe1e010) at ../iocore/eventsystem/I_Lock.h:380
#1  0x000000000053d518 in MutexLock::MutexLock (this=0x2b85a571f1d0, am=0x0, t=0x2b859fe1e010) at ../iocore/eventsystem/I_Lock.h:447
#2  0x0000000000653944 in Http2Stream::initiating_close (this=0x2b86a10995c0) at Http2Stream.cc:367
#3  0x0000000000650210 in Http2ConnectionState::delete_stream (this=0x2b85ac54ce28, stream=0x2b86a10995c0)
    at Http2ConnectionState.cc:879
#4  0x000000000065097d in Http2ConnectionState::send_data_frame (this=0x2b85ac54ce28, stream=0x2b86a10995c0)
    at Http2ConnectionState.cc:986
#5  0x000000000064f441 in rcv_window_update_frame (cs=..., cstate=..., frame=...) at Http2ConnectionState.cc:591
#6  0x000000000064fcdb in Http2ConnectionState::main_event_handler (this=0x2b85ac54ce28, event=2253, edata=0x2b85a5721610)
    at Http2ConnectionState.cc:753
#7  0x0000000000515a58 in Continuation::handleEvent (this=0x2b85ac54ce28, event=2253, data=0x2b85a5721610)
    at ../iocore/eventsystem/I_Continuation.h:150
#8  0x0000000000649ef9 in send_connection_event (cont=0x2b85ac54ce28, event=2253, edata=0x2b85a5721610) at Http2ClientSession.cc:61
#9  0x000000000064c247 in Http2ClientSession::state_complete_frame_read (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:482
#10 0x000000000064b009 in Http2ClientSession::main_event_handler (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:304
#11 0x0000000000515a58 in Continuation::handleEvent (this=0x2b85ac54cbf0, event=100, data=0x2b86fc70bff0)
    at ../iocore/eventsystem/I_Continuation.h:150
#12 0x000000000064bf8f in Http2ClientSession::state_start_frame_read (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:455
#13 0x000000000064b009 in Http2ClientSession::main_event_handler (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:304
#14 0x0000000000515a58 in Continuation::handleEvent (this=0x2b85ac54cbf0, event=100, data=0x2b86fc70bff0)
    at ../iocore/eventsystem/I_Continuation.h:150
#15 0x000000000064c2b8 in Http2ClientSession::state_complete_frame_read (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:487
#16 0x000000000064b009 in Http2ClientSession::main_event_handler (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:304
#17 0x0000000000515a58 in Continuation::handleEvent (this=0x2b85ac54cbf0, event=100, data=0x2b86fc70bff0)
    at ../iocore/eventsystem/I_Continuation.h:150
#18 0x000000000064bf8f in Http2ClientSession::state_start_frame_read (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:455
#19 0x000000000064b009 in Http2ClientSession::main_event_handler (this=0x2b85ac54cbf0, event=100, edata=0x2b86fc70bff0)
    at Http2ClientSession.cc:304
#20 0x0000000000515a58 in Continuation::handleEvent (this=0x2b85ac54cbf0, event=100, data=0x2b86fc70bff0)
    at ../iocore/eventsystem/I_Continuation.h:150
#21 0x000000000078679d in read_signal_and_update (event=100, vc=0x2b86fc70bed0) at UnixNetVConnection.cc:148
#22 0x00000000007895ba in UnixNetVConnection::readSignalAndUpdate (this=0x2b86fc70bed0, event=100) at UnixNetVConnection.cc:1013
#23 0x000000000076de67 in SSLNetVConnection::net_read_io (this=0x2b86fc70bed0, nh=0x2b859fe21d60, lthread=0x2b859fe1e010)
    at SSLNetVConnection.cc:576
---Type <return> to continue, or q <return> to quit---
#24 0x0000000000780011 in NetHandler::waitForActivity (this=0x2b859fe21d60, timeout=60000000) at UnixNet.cc:547
#25 0x00000000007a7c69 in EThread::execute_regular (this=0x2b859fe1e010) at UnixEThread.cc:266
#26 0x00000000007a7dac in EThread::execute (this=0x2b859fe1e010) at UnixEThread.cc:304
#27 0x00000000007a6965 in spawn_thread_internal (a=0x112f520) at Thread.cc:85
#28 0x00002b859d8deaa1 in start_thread () from /lib64/libpthread.so.0
#29 0x00000031d68e893d in clone () from /lib64/libc.so.6

I think the issue was that I removed a schedule_immediate _EOS to do the Http2Stream destroy in an earlier fi and instead did the destroy inline. But that class does not include recursion logic to delay the delete until the stack is unbound. By sending the event again it unwinds the stack before doing the destroy of Http2Stream.