[TS-2355] ATS 4.0.x crashes when using OpenSSL 1.0.1e

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.1, 4.1.2
• Fix Version/s: 4.2.0
• Component/s: SSL
• Labels: T
• Assignee: Bryan Call
• Reporter: David Carlin
• Created: 15/Nov/13 22:39
• Updated: 27/Feb/14 22:17
• Resolved: 16/Dec/13 23:12

Description

I upgraded some 4.0.1 and 4.0.2 hosts from OpenSSL 1.0.0 to 1.0.1e which is supposed to be ABI compatible. I see this crash about 10 times in a given 24 hour period.

I'm interested in OpenSSL 1.0.1e as there is a CPU usage improvement in my tests, and for TLS 1.2 support.

I came across this squid bug with a very similar backtrace. The OpenSSL RT ticket says

"I have discussed this situation with some Squid developers and we decided - after SSL error 1408F10B calling standard/raw read() instead of SSL_read() for empty socket buffer and this patch stopped crash Squid."

http://rt.openssl.org/Ticket/Display.html?id=3128&user=guest&pass=guest

#0  0x0000003f842e7154 in EVP_DigestFinal_ex () from /usr/lib64/libcrypto.so.10
#1  0x0000003f84636263 in tls1_final_finish_mac () from /usr/lib64/libssl.so.10
#2  0x0000003f8462ad62 in ssl3_do_change_cipher_spec () from /usr/lib64/libssl.so.10
#3  0x0000003f8462c7f7 in ssl3_read_bytes () from /usr/lib64/libssl.so.10
#4  0x0000003f8462d5e2 in ssl3_get_message () from /usr/lib64/libssl.so.10
#5  0x0000003f8461da1c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10
#6  0x0000003f84621e78 in ssl3_accept () from /usr/lib64/libssl.so.10
#7  0x00000000006711aa in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aadd0024300,
    err=@0x2aacab940c5c) at SSLNetVConnection.cc:488
#8  0x0000000000672b77 in SSLNetVConnection::sslStartHandShake (this=0x2aadd0024300,
    event=<value optimized out>, err=@0x2aacab940c5c) at SSLNetVConnection.cc:470
#9  0x0000000000671dd2 in SSLNetVConnection::net_read_io (this=0x2aadd0024300, nh=
    0x2aacaa02cbf0, lthread=0x2aacaa029010) at SSLNetVConnection.cc:217
#10 0x000000000067b8c2 in NetHandler::mainNetEvent (this=0x2aacaa02cbf0,
    event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
#11 0x00000000006a335f in handleEvent (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
    at I_Continuation.h:146
#12 EThread::process_event (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
    at UnixEThread.cc:141
#13 0x00000000006a3d43 in EThread::execute (this=0x2aacaa029010) at UnixEThread.cc:265
#14 0x00000000006a21fa in spawn_thread_internal (a=0x143ec30) at Thread.cc:88
#15 0x00002aaca05b9851 in start_thread () from /lib64/libpthread.so.0
#16 0x000000324f0e890d in clone () from /lib64/libc.so.6

NOTE: Traffic Server received Sig 11: Segmentation fault
/home/y/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0x324f40f500)[0x2b523d64e500]
/usr/lib64/libcrypto.so.10(EVP_DigestFinal_ex+0x24)[0x3f842e7154]
/usr/lib64/libssl.so.10(tls1_final_finish_mac+0x233)[0x3f84636263]
/usr/lib64/libssl.so.10(ssl3_do_change_cipher_spec+0x72)[0x3f8462ad62]
/usr/lib64/libssl.so.10(ssl3_read_bytes+0xb57)[0x3f8462c7f7]
/usr/lib64/libssl.so.10(ssl3_get_message+0x222)[0x3f8462d5e2]
/usr/lib64/libssl.so.10(ssl3_get_cert_verify+0x6c)[0x3f8461da1c]
/usr/lib64/libssl.so.10(ssl3_accept+0x788)[0x3f84621e78]
/home/y/bin/traffic_server(SSLNetVConnection::sslServerHandShakeEvent(int&)+0x2a)[0x6711aa]
/home/y/bin/traffic_server(SSLNetVConnection::sslStartHandShake(int, int&)+0x37)[0x672b77]
/home/y/bin/traffic_server(SSLNetVConnection::net_read_io(NetHandler*, EThread*)+0x1f2)[0x671dd2]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1f2)[0x67b8c2]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0x8f)[0x6a335f]
/home/y/bin/traffic_server(EThread::execute()+0x4a3)[0x6a3d43]
/home/y/bin/traffic_server[0x6a21fa]
/lib64/libpthread.so.0(+0x324f407851)[0x2b523d646851]
/lib64/libc.so.6(clone+0x6d)[0x324f0e890d]

Issue Links

• relates to: TS-2593 HTTPS to origin fails on CentOS6.x