[TS-2776] Core dump inside openssl library

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 4.2.2, 5.0.0
• Component/s: SPDY, SSL
• Labels: spdy yahoo
• Assignee: Bryan Call
• Reporter: Sudheer Vinukonda
• Created: 02/May/14 16:20
• Updated: 18/Dec/14 05:12
• Resolved: 03/Jun/14 19:07

Description

During production testing of SPDY (w/ ATS compiled from master repo), noticed the below core from open ssl library. This core showed up after fixing a few other core dumps and memory leaks (refer TS-2742, TS-2743, TS-2750, TS-2765, TS-2767 etc) and happened once so far, after running stable for 30+ hours on a production host.

[example_prep.sh] Checking/Moving old cores...
[TrafficServer] using root directory '/home/y'
[TrafficManager] ==> Cleaning up and reissuing signal #15
[TrafficManager] ==> signal #15
[E. Mgmt] log ==> [TrafficManager] using root directory '/home/y'
[example_prep.sh] Checking/Moving old cores...
[TrafficServer] using root directory '/home/y'
NOTE: Traffic Server received Sig 11: Segmentation fault
/home/y/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0x329720f500)[0x2b7dff1f6500]
/usr/lib64/libssl.so.10[0x331622b2e7]
/usr/lib64/libssl.so.10(ssl3_write_bytes+0x75)[0x331622b905]
/home/y/bin/traffic_server(_ZN17SSLNetVConnection21load_buffer_and_writeElRlS0_R17MIOBufferAccessorRi+0xdf)[0x6db57f]
/home/y/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x418)[0x6ef488]
/home/y/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x283)[0x6e3ee3]
/home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x7109ff]
/home/y/bin/traffic_server(_ZN7EThread7executeEv+0x43b)[0x71122b]
/home/y/bin/traffic_server[0x70fdaa]
/lib64/libpthread.so.0(+0x3297207851)[0x2b7dff1ee851]
/lib64/libc.so.6(clone+0x6d)[0x3296ee890d]
[E. Mgmt] log ==> [TrafficManager] using root directory '/home/y'
[example_prep.sh] Checking/Moving old cores...
[TrafficServer] using root directory '/home/y'

gdb output below:

(gdb) bt
#0  0x000000331622b2e7 in ?? () from /usr/lib64/libssl.so.10
#1  0x000000331622b905 in ssl3_write_bytes () from /usr/lib64/libssl.so.10
#2  0x00000000006db57f in do_SSL_write (this=0x2b7f42c7ca70, towrite=207693, wattempted=@0x2b7e111aec28, total_wrote=@0x2b7e111aec30, buf=<value optimized out>, needs=@0x2b7e111aec38)
    at SSLNetVConnection.cc:90
#3  SSLNetVConnection::load_buffer_and_write (this=0x2b7f42c7ca70, towrite=207693, wattempted=@0x2b7e111aec28, total_wrote=@0x2b7e111aec30, buf=<value optimized out>, needs=@0x2b7e111aec38)
    at SSLNetVConnection.cc:399
#4  0x00000000006ef488 in write_to_net_io (nh=0x2b7e0f89ac10, vc=0x2b7f42c7ca70, thread=0x2b7e0f897010) at UnixNetVConnection.cc:527
#5  0x00000000006e3ee3 in NetHandler::mainNetEvent (this=0x2b7e0f89ac10, event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:400
#6  0x00000000007109ff in handleEvent (this=0x2b7e0f897010, e=0x31ae0d0, calling_code=5) at I_Continuation.h:146
#7  EThread::process_event (this=0x2b7e0f897010, e=0x31ae0d0, calling_code=5) at UnixEThread.cc:145
#8  0x000000000071122b in EThread::execute (this=0x2b7e0f897010) at UnixEThread.cc:269
#9  0x000000000070fdaa in spawn_thread_internal (a=0x2f8fbe0) at Thread.cc:88
#10 0x00002b7dff1ee851 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003296ee890d in clone () from /lib64/libc.so.6
(gdb) up
#1  0x000000331622b905 in ssl3_write_bytes () from /usr/lib64/libssl.so.10
(gdb) up
#2  0x00000000006db57f in do_SSL_write (this=0x2b7f42c7ca70, towrite=207693, wattempted=@0x2b7e111aec28, total_wrote=@0x2b7e111aec30, buf=<value optimized out>, needs=@0x2b7e111aec38)
    at SSLNetVConnection.cc:90
90	SSLNetVConnection.cc: No such file or directory.
	in SSLNetVConnection.cc
(gdb) 

Issue Links

• is duplicated by: TS-2817 ATS crash in SSLNetVConnection::load_buffer_and_write