Traffic Server
  1. Traffic Server
  2. TS-2355

ATS 4.0.x crashes when using OpenSSL 1.0.1e

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.1, 4.1.2
    • Fix Version/s: 4.2.0
    • Component/s: SSL
    • Labels:

      Description

      I upgraded some 4.0.1 and 4.0.2 hosts from OpenSSL 1.0.0 to 1.0.1e which is supposed to be ABI compatible. I see this crash about 10 times in a given 24 hour period.

      I'm interested in OpenSSL 1.0.1e as there is a CPU usage improvement in my tests, and for TLS 1.2 support.

      I came across this squid bug with a very similar backtrace. The OpenSSL RT ticket says

      "I have discussed this situation with some Squid developers and we decided - after SSL error 1408F10B calling standard/raw read() instead of SSL_read() for empty socket buffer and this patch stopped crash Squid."

      http://rt.openssl.org/Ticket/Display.html?id=3128&user=guest&pass=guest

      #0  0x0000003f842e7154 in EVP_DigestFinal_ex () from /usr/lib64/libcrypto.so.10
      #1  0x0000003f84636263 in tls1_final_finish_mac () from /usr/lib64/libssl.so.10
      #2  0x0000003f8462ad62 in ssl3_do_change_cipher_spec () from /usr/lib64/libssl.so.10
      #3  0x0000003f8462c7f7 in ssl3_read_bytes () from /usr/lib64/libssl.so.10
      #4  0x0000003f8462d5e2 in ssl3_get_message () from /usr/lib64/libssl.so.10
      #5  0x0000003f8461da1c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10
      #6  0x0000003f84621e78 in ssl3_accept () from /usr/lib64/libssl.so.10
      #7  0x00000000006711aa in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aadd0024300,
          err=@0x2aacab940c5c) at SSLNetVConnection.cc:488
      #8  0x0000000000672b77 in SSLNetVConnection::sslStartHandShake (this=0x2aadd0024300,
          event=<value optimized out>, err=@0x2aacab940c5c) at SSLNetVConnection.cc:470
      #9  0x0000000000671dd2 in SSLNetVConnection::net_read_io (this=0x2aadd0024300, nh=
          0x2aacaa02cbf0, lthread=0x2aacaa029010) at SSLNetVConnection.cc:217
      #10 0x000000000067b8c2 in NetHandler::mainNetEvent (this=0x2aacaa02cbf0,
          event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
      #11 0x00000000006a335f in handleEvent (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
          at I_Continuation.h:146
      #12 EThread::process_event (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
          at UnixEThread.cc:141
      #13 0x00000000006a3d43 in EThread::execute (this=0x2aacaa029010) at UnixEThread.cc:265
      #14 0x00000000006a21fa in spawn_thread_internal (a=0x143ec30) at Thread.cc:88
      #15 0x00002aaca05b9851 in start_thread () from /lib64/libpthread.so.0
      #16 0x000000324f0e890d in clone () from /lib64/libc.so.6
      
      NOTE: Traffic Server received Sig 11: Segmentation fault
      /home/y/bin/traffic_server - STACK TRACE:
      /lib64/libpthread.so.0(+0x324f40f500)[0x2b523d64e500]
      /usr/lib64/libcrypto.so.10(EVP_DigestFinal_ex+0x24)[0x3f842e7154]
      /usr/lib64/libssl.so.10(tls1_final_finish_mac+0x233)[0x3f84636263]
      /usr/lib64/libssl.so.10(ssl3_do_change_cipher_spec+0x72)[0x3f8462ad62]
      /usr/lib64/libssl.so.10(ssl3_read_bytes+0xb57)[0x3f8462c7f7]
      /usr/lib64/libssl.so.10(ssl3_get_message+0x222)[0x3f8462d5e2]
      /usr/lib64/libssl.so.10(ssl3_get_cert_verify+0x6c)[0x3f8461da1c]
      /usr/lib64/libssl.so.10(ssl3_accept+0x788)[0x3f84621e78]
      /home/y/bin/traffic_server(SSLNetVConnection::sslServerHandShakeEvent(int&)+0x2a)[0x6711aa]
      /home/y/bin/traffic_server(SSLNetVConnection::sslStartHandShake(int, int&)+0x37)[0x672b77]
      /home/y/bin/traffic_server(SSLNetVConnection::net_read_io(NetHandler*, EThread*)+0x1f2)[0x671dd2]
      /home/y/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1f2)[0x67b8c2]
      /home/y/bin/traffic_server(EThread::process_event(Event*, int)+0x8f)[0x6a335f]
      /home/y/bin/traffic_server(EThread::execute()+0x4a3)[0x6a3d43]
      /home/y/bin/traffic_server[0x6a21fa]
      /lib64/libpthread.so.0(+0x324f407851)[0x2b523d646851]
      /lib64/libc.so.6(clone+0x6d)[0x324f0e890d]
      
      1. ts2355.diff
        3 kB
        Ron Barber
      2. ts2355.diff
        3 kB
        Ron Barber

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          30d 23h 19m 1 Ron Barber 16/Dec/13 21:59
          Patch Available Patch Available Closed Closed
          1h 13m 1 Bryan Call 16/Dec/13 23:12
          Hide
          Leif Hedstrom added a comment -

          For posterity, I modified the default here to keep the defaults at "on".

          Show
          Leif Hedstrom added a comment - For posterity, I modified the default here to keep the defaults at "on".
          Leif Hedstrom made changes -
          Link This issue relates to TS-2593 [ TS-2593 ]
          Hide
          Leif Hedstrom added a comment -

          Ron Barber Should we update the wiki as well with this information and CVE ?

          https://cwiki.apache.org/confluence/display/TS/What%27s+new+in+v4.2.x

          Show
          Leif Hedstrom added a comment - Ron Barber Should we update the wiki as well with this information and CVE ? https://cwiki.apache.org/confluence/display/TS/What%27s+new+in+v4.2.x
          Hide
          Ron Barber added a comment -

          This is now covered by CVE-2013-6449 and fixed in 1.0.1f:

          http://www.openssl.org/news/vulnerabilities.html#2013-6449

          Show
          Ron Barber added a comment - This is now covered by CVE-2013-6449 and fixed in 1.0.1f: http://www.openssl.org/news/vulnerabilities.html#2013-6449
          Zhao Yongming made changes -
          Labels T
          Hide
          Ron Barber added a comment -

          The patch to OpenSSL has been committed to their repository and will appear in OpenSSL 1.0.1f and later. Expected availability is by end of January 2014.

          I have been told that OpenSSL v1.0.2 should not have this problem.

          Show
          Ron Barber added a comment - The patch to OpenSSL has been committed to their repository and will appear in OpenSSL 1.0.1f and later. Expected availability is by end of January 2014. I have been told that OpenSSL v1.0.2 should not have this problem.
          Hide
          Ron Barber added a comment -

          I have been working with the OpenSSL community offline as they try to find the
          root cause of the crash as it is apparently a potential DOS vulnerability. The
          first patch they provided as part of the bug was just a band aid. The second
          patch actually attempts to fix the root cause and I tried it in production for
          4 hours and did not experience a crash (normally crashes in less than 1 hour). I have inquired about whether the
          patch would be back ported to OpenSSL 1.0.1 and will update this bug again once
          I know which version(s) of OpenSSL have the fix.

          Show
          Ron Barber added a comment - I have been working with the OpenSSL community offline as they try to find the root cause of the crash as it is apparently a potential DOS vulnerability. The first patch they provided as part of the bug was just a band aid. The second patch actually attempts to fix the root cause and I tried it in production for 4 hours and did not experience a crash (normally crashes in less than 1 hour). I have inquired about whether the patch would be back ported to OpenSSL 1.0.1 and will update this bug again once I know which version(s) of OpenSSL have the fix.
          Bryan Call made changes -
          Status Patch Available [ 10002 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Hide
          Bryan Call added a comment -

          Patch added to have configuration options to turn on/off TLS 1.1 and 1.2. By default TLS 1.2 is off, for now. We will keep track of the openssl ticket and will have to change the default to enable TLS 1.2 later after the issue has been fixed:

          openssl ticket:
          http://rt.openssl.org/Ticket/Display.html?id=3200

          Show
          Bryan Call added a comment - Patch added to have configuration options to turn on/off TLS 1.1 and 1.2. By default TLS 1.2 is off, for now. We will keep track of the openssl ticket and will have to change the default to enable TLS 1.2 later after the issue has been fixed: openssl ticket: http://rt.openssl.org/Ticket/Display.html?id=3200
          Hide
          Ron Barber added a comment -

          Version is a left over which I failed to remove after removing references from the code. Sorry.

          Show
          Ron Barber added a comment - Version is a left over which I failed to remove after removing references from the code. Sorry.
          Hide
          ASF subversion and git services added a comment -

          Commit 84f92083cd3aa46f4095eb00463dbace1cc2708c in branch refs/heads/master from Bryan Call
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=84f9208 ]

          TS-2355: ATS 4.0.x crashes when using OpenSSL 1.0.1e

          Show
          ASF subversion and git services added a comment - Commit 84f92083cd3aa46f4095eb00463dbace1cc2708c in branch refs/heads/master from Bryan Call [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=84f9208 ] TS-2355 : ATS 4.0.x crashes when using OpenSSL 1.0.1e
          Hide
          ASF subversion and git services added a comment -

          Commit 2a979548dbf17dea5fbeb43e79116b4c3dcf4a6e in branch refs/heads/master from Ron Barber
          [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=2a97954 ]

          TS-2355: ATS 4.0.x crashes when using OpenSSL 1.0.1e

          Show
          ASF subversion and git services added a comment - Commit 2a979548dbf17dea5fbeb43e79116b4c3dcf4a6e in branch refs/heads/master from Ron Barber [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=2a97954 ] TS-2355 : ATS 4.0.x crashes when using OpenSSL 1.0.1e
          Hide
          James Peach added a comment -

          What's the OpenSSL version number for?

          Show
          James Peach added a comment - What's the OpenSSL version number for?
          Ron Barber made changes -
          Attachment ts2355.diff [ 12618988 ]
          Hide
          Ron Barber added a comment -

          Move #defines up in P_SSLUtils.h
          Removed inconsistent blank lines in SSLConfig.cc

          Show
          Ron Barber added a comment - Move #defines up in P_SSLUtils.h Removed inconsistent blank lines in SSLConfig.cc
          Hide
          Bryan Call added a comment -

          Talked to this with Ron over IM:

          1. remove spaces at the end of the lines
          2. move defines after the include of the openssl headers
          3. have the same line spacing as the code in SSLConfig.cc

          Show
          Bryan Call added a comment - Talked to this with Ron over IM: 1. remove spaces at the end of the lines 2. move defines after the include of the openssl headers 3. have the same line spacing as the code in SSLConfig.cc
          Bryan Call made changes -
          Assignee Bryan Call [ bcall ]
          Hide
          Ron Barber added a comment -

          Patch adds configuration items to disable TLS 1.1 and 1.2. By default TLS v1.2 is disabled (if the configuration value is not present in records.config).

          Example/default values:
          CONFIG proxy.config.ssl.TLSv1_1 INT 1
          CONFIG proxy.config.ssl.TLSv1_2 INT 0

          Show
          Ron Barber added a comment - Patch adds configuration items to disable TLS 1.1 and 1.2. By default TLS v1.2 is disabled (if the configuration value is not present in records.config). Example/default values: CONFIG proxy.config.ssl.TLSv1_1 INT 1 CONFIG proxy.config.ssl.TLSv1_2 INT 0
          Hide
          Ron Barber added a comment -
          Show
          Ron Barber added a comment - Patch attached. Better OpenSSL link: http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3200
          Ron Barber made changes -
          Attachment ts2355.diff [ 12618983 ]
          Ron Barber made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ron Barber added a comment -
          Show
          Ron Barber added a comment - OpenSSL ticket created: http://rt.openssl.org/Ticket/Display.html?id=3200
          Hide
          Ron Barber added a comment -

          I have done some "debugging" on this although I know little about openssl. I have come to the conclusion that the issue lies withing openssl 1.0.1e (and 1.0.1.a) and does not exist with 1.0.0k. Specifically, I have modified TS to support "CONFIG proxy.config.ssl.TLSv1_2" which if set to 0 disables TLS 1.2 support. With TLS 1.2 disabled, TS did not crash after running over 24 hours. I crashes in less than 1 hour (during peak) with TLS 1.2 enabled.

          Two thoughts:
          1. This ticket should be a feature request to add 2 configuration options which control enablement of TLS 1.1 and TLS 1.2 (similar to the existing TLSv1 config):
          "CONFIG proxy.config.ssl.TLSv1 INT 1" ** EXISTING CONFIG OPTION **
          "CONFIG proxy.config.ssl.TLSv1_1 INT 1"
          "CONFIG proxy.config.ssl.TLSv1_2 INT 1"

          2. Create a ticket/report w/openssl community for this issue. Here is a sample debug session (we are running RHEL6):

          Program terminated with signal 11, Segmentation fault.
          #0  0x00002aed38e036b1 in EVP_DigestFinal_ex (ctx=0x2aed482007d0, md=0x2aed48200750 "", size=0x2aed48200804) at digest.c:271
          271     digest.c: No such file or directory.
                  in digest.c
          Missing separate debuginfos, use: debuginfo-install expat-2.0.1-11.el6_2.x86_64 glibc-2.12-1.107.el6.x86_64 hwloc-1.5-1.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libevent-1.4.13-4.el6.x86_64 libgcc-4.4.7-3.el6.x86_64 libstdc++-4.4.7-3.el6.x86_64 libxml2-2.7.6-12.el6_4.1.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 numactl-2.0.7-6.el6.x86_64 openssl-1.0.0-27.el6.x86_64 pciutils-libs-3.1.10-2.el6.x86_64 pcre-7.8-6.el6.x86_64 tcl-8.5.7-6.el6.x86_64 xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64
          (gdb) where
          #0  0x00002aed38e036b1 in EVP_DigestFinal_ex (ctx=0x2aed482007d0, md=0x2aed48200750 "", size=0x2aed48200804) at digest.c:271
          #1  0x00002aed38ab0c0b in tls1_final_finish_mac (s=0x2aedd06d7990, str=0x2aed38ad7869 "client finished", slen=15, out=0x2aedd04b0b24 "") at t1_enc.c:926
          #2  0x00002aed38aa413c in ssl3_do_change_cipher_spec (s=0x2aedd06d7990) at s3_pkt.c:1462
          #3  0x00002aed38aa3c58 in ssl3_read_bytes (s=0x2aedd06d7990, type=22, buf=0x2aedd0388400 "\020", len=4, peek=0) at s3_pkt.c:1306
          #4  0x00002aed38aa5068 in ssl3_get_message (s=0x2aedd06d7990, st1=8608, stn=8609, mt=-1, max=516, ok=0x2aed48200a9c) at s3_both.c:451
          #5  0x00002aed38a93ed7 in ssl3_get_cert_verify (s=0x2aedd06d7990) at s3_srvr.c:2924
          #6  0x00002aed38a8f25c in ssl3_accept (s=0x2aedd06d7990) at s3_srvr.c:677
          #7  0x00002aed38ac131c in SSL_accept (s=0x2aedd06d7990) at ssl_lib.c:940
          #8  0x00000000006710ba in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aedc0129cb0, err=@0x2aed48200d1c) at SSLNetVConnection.cc:488
          #9  0x0000000000672977 in SSLNetVConnection::sslStartHandShake (this=0x2aedc0129cb0, event=<value optimized out>, err=@0x2aed48200d1c) at SSLNetVConnection.cc:470
          #10 0x0000000000671bd2 in SSLNetVConnection::net_read_io (this=0x2aedc0129cb0, nh=0x2aed42834bf0, lthread=0x2aed42831010) at SSLNetVConnection.cc:217
          #11 0x000000000067b6b2 in NetHandler::mainNetEvent (this=0x2aed42834bf0, event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
          #12 0x00000000006a314f in handleEvent (this=0x2aed42831010, e=0x113cc70, calling_code=5) at I_Continuation.h:146
          #13 EThread::process_event (this=0x2aed42831010, e=0x113cc70, calling_code=5) at UnixEThread.cc:141
          #14 0x00000000006a3b33 in EThread::execute (this=0x2aed42831010) at UnixEThread.cc:265
          #15 0x00000000006a1fea in spawn_thread_internal (a=0x1349630) at Thread.cc:88
          #16 0x00002aed3934d851 in start_thread () from /lib64/libpthread.so.0
          #17 0x000000324f0e890d in clone () from /lib64/libc.so.6
          (gdb) f 7
          #7  0x00002aed38ac131c in SSL_accept (s=0x2aedd06d7990) at ssl_lib.c:940
          940     ssl_lib.c: No such file or directory.
                  in ssl_lib.c
          (gdb) print *s
          $1 = {version = 769, type = 8192, method = 0x2aed38ce6e00, rbio = 0x2aedd024f760, wbio = 0x2aedd006a7e0, bbio = 0x2aedd006a7e0, rwstate = 1, in_handshake = 1, handshake_func = 0x2aed38a8e41e <ssl3_accept>, server = 1, new_session = 0, 
            quiet_shutdown = 1, shutdown = 0, state = 8608, rstate = 240, init_buf = 0x2aedd055b2d0, init_msg = 0x2aedd0388404, init_num = 0, init_off = 0, packet = 0x2aee3816fbf3 "\024\003\001", packet_length = 0, s2 = 0x0, s3 = 0x2aedd04b0810, 
            d1 = 0x0, read_ahead = 0, msg_callback = 0, msg_callback_arg = 0x0, hit = 0, param = 0x2aedd00060e0, cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx = 0x2aedd0697ce0, read_hash = 0x2aedd03399a0, expand = 0x0, 
            enc_write_ctx = 0x0, write_hash = 0x0, compress = 0x0, cert = 0x2aedd00e4030, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session = 0x2aedd01cc080, generate_session_id = 0, verify_mode = 0, verify_callback = 0, 
            info_callback = 0, error = 0, error_code = 0, psk_client_callback = 0, psk_server_callback = 0, ctx = 0x1344430, debug = 0, verify_result = 0, ex_data = {sk = 0x2aedd033a6c0, dummy = 0}, client_CA = 0x0, references = 1, options = 21102596, 
            mode = 0, max_cert_list = 102400, first_packet = 0, client_version = 771, max_send_fragment = 16384, tlsext_debug_cb = 0, tlsext_debug_arg = 0x0, tlsext_hostname = 0x0, servername_done = 1, tlsext_status_type = -1, 
            tlsext_status_expected = 0, tlsext_ocsp_ids = 0x0, tlsext_ocsp_exts = 0x0, tlsext_ocsp_resp = 0x0, tlsext_ocsp_resplen = -1, tlsext_ticket_expected = 1, tlsext_ecpointformatlist_length = 0, tlsext_ecpointformatlist = 0x0, 
            tlsext_ellipticcurvelist_length = 0, tlsext_ellipticcurvelist = 0x0, tlsext_opaque_prf_input = 0x0, tlsext_opaque_prf_input_len = 0, tlsext_session_ticket = 0x0, tls_session_ticket_ext_cb = 0, tls_session_ticket_ext_cb_arg = 0x0, 
            tls_session_secret_cb = 0, tls_session_secret_cb_arg = 0x0, initial_ctx = 0x1344430, next_proto_negotiated = 0x0, next_proto_negotiated_len = 0 '\000', srtp_profiles = 0x0, srtp_profile = 0x0, tlsext_heartbeat = 0, tlsext_hb_pending = 0, 
            tlsext_hb_seq = 0, renegotiate = 2, srp_ctx = {SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0, SRP_verify_param_callback = 0, SRP_give_srp_client_pwd_callback = 0, login = 0x0, N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, 
              b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0}}
          (gdb) print *s->ctx
          $2 = {method = 0x2aed38ce6700, cipher_list = 0x1345360, cipher_list_by_id = 0x1345240, cert_store = 0x13449a0, sessions = 0x1344850, session_cache_size = 20480, session_cache_head = 0x2aede82fe200, session_cache_tail = 0x2aee080010b0, 
            session_cache_mode = 2, session_timeout = 300, new_session_cb = 0, remove_session_cb = 0, get_session_cb = 0, stats = {sess_connect = 0, sess_connect_renegotiate = 0, sess_connect_good = 0, sess_accept = 12625, sess_accept_renegotiate = 0, 
              sess_accept_good = 12321, sess_miss = 1549, sess_timeout = 0, sess_cache_full = 0, sess_hit = 3043, sess_cb_hit = 0}, references = 8057, app_verify_callback = 0, app_verify_arg = 0x0, default_passwd_callback = 0, 
            default_passwd_callback_userdata = 0x0, client_cert_cb = 0, app_gen_cookie_cb = 0, app_verify_cookie_cb = 0, ex_data = {sk = 0x0, dummy = 0}, rsa_md5 = 0x2aed390f7c20, md5 = 0x2aed390f7c20, sha1 = 0x2aed390f7d20, extra_certs = 0x1346ca0, 
            comp_methods = 0x1342ce0, info_callback = 0, client_CA = 0x1345060, options = 21102596, mode = 0, max_cert_list = 102400, cert = 0x1344720, read_ahead = 0, msg_callback = 0, msg_callback_arg = 0x0, verify_mode = 0, sid_ctx_length = 0, 
            sid_ctx = '\000' <repeats 31 times>, default_verify_callback = 0, generate_session_id = 0, param = 0x1345020, quiet_shutdown = 1, max_send_fragment = 16384, client_cert_engine = 0x0, tlsext_servername_callback = 0x675550
               <ssl_servername_callback(SSL*, int*, void*)>, tlsext_servername_arg = 0x1343120, tlsext_tick_key_name = "\r(T[\177\025\267\216\326\213ω:\277a)", tlsext_tick_hmac_key = "]wEz9.Cȕc\237\002_--o", 
            tlsext_tick_aes_key = "@7\333a\026cf\274\312\346\273]m\344\217A", tlsext_ticket_key_cb = 0, tlsext_status_cb = 0, tlsext_status_arg = 0x0, tlsext_opaque_prf_input_callback = 0, tlsext_opaque_prf_input_callback_arg = 0x0, 
            psk_identity_hint = 0x0, psk_client_callback = 0, psk_server_callback = 0, freelist_max_len = 32, wbuf_freelist = 0x13451f0, rbuf_freelist = 0x13451d0, srp_ctx = {SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0, 
              SRP_verify_param_callback = 0, SRP_give_srp_client_pwd_callback = 0, login = 0x0, N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0}, next_protos_advertised_cb = 0, 
            next_protos_advertised_cb_arg = 0x0, next_proto_select_cb = 0, next_proto_select_cb_arg = 0x0, srtp_profiles = 0x0}
          (gdb) f 1
          #1  0x00002aed38ab0c0b in tls1_final_finish_mac (s=0x2aedd06d7990, str=0x2aed38ad7869 "client finished", slen=15, out=0x2aedd04b0b24 "") at t1_enc.c:926
          926     t1_enc.c: No such file or directory.
                  in t1_enc.c
          (gdb) print *s->ctx
          $3 = {method = 0x2aed38ce6700, cipher_list = 0x1345360, cipher_list_by_id = 0x1345240, cert_store = 0x13449a0, sessions = 0x1344850, session_cache_size = 20480, session_cache_head = 0x2aede82fe200, session_cache_tail = 0x2aee080010b0, 
            session_cache_mode = 2, session_timeout = 300, new_session_cb = 0, remove_session_cb = 0, get_session_cb = 0, stats = {sess_connect = 0, sess_connect_renegotiate = 0, sess_connect_good = 0, sess_accept = 12625, sess_accept_renegotiate = 0, 
              sess_accept_good = 12321, sess_miss = 1549, sess_timeout = 0, sess_cache_full = 0, sess_hit = 3043, sess_cb_hit = 0}, references = 8057, app_verify_callback = 0, app_verify_arg = 0x0, default_passwd_callback = 0, 
            default_passwd_callback_userdata = 0x0, client_cert_cb = 0, app_gen_cookie_cb = 0, app_verify_cookie_cb = 0, ex_data = {sk = 0x0, dummy = 0}, rsa_md5 = 0x2aed390f7c20, md5 = 0x2aed390f7c20, sha1 = 0x2aed390f7d20, extra_certs = 0x1346ca0, 
            comp_methods = 0x1342ce0, info_callback = 0, client_CA = 0x1345060, options = 21102596, mode = 0, max_cert_list = 102400, cert = 0x1344720, read_ahead = 0, msg_callback = 0, msg_callback_arg = 0x0, verify_mode = 0, sid_ctx_length = 0, 
            sid_ctx = '\000' <repeats 31 times>, default_verify_callback = 0, generate_session_id = 0, param = 0x1345020, quiet_shutdown = 1, max_send_fragment = 16384, client_cert_engine = 0x0, 
            tlsext_servername_callback = 0x675550 <ssl_servername_callback(SSL*, int*, void*)>, tlsext_servername_arg = 0x1343120, tlsext_tick_key_name = "\r(T[\177\025\267\216\326\213ω:\277a)", tlsext_tick_hmac_key = "]wEz9.Cȕc\237\002_--o", 
            tlsext_tick_aes_key = "@7\333a\026cf\274\312\346\273]m\344\217A", tlsext_ticket_key_cb = 0, tlsext_status_cb = 0, tlsext_status_arg = 0x0, tlsext_opaque_prf_input_callback = 0, tlsext_opaque_prf_input_callback_arg = 0x0, 
            psk_identity_hint = 0x0, psk_client_callback = 0, psk_server_callback = 0, freelist_max_len = 32, wbuf_freelist = 0x13451f0, rbuf_freelist = 0x13451d0, srp_ctx = {SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0, 
              SRP_verify_param_callback = 0, SRP_give_srp_client_pwd_callback = 0, login = 0x0, N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0}, next_protos_advertised_cb = 0, 
            next_protos_advertised_cb_arg = 0x0, next_proto_select_cb = 0, next_proto_select_cb_arg = 0x0, srtp_profiles = 0x0}
          (gdb) f 1
          #1  0x00002aed38ab0c0b in tls1_final_finish_mac (s=0x2aedd06d7990, str=0x2aed38ad7869 "client finished", slen=15, out=0x2aedd04b0b24 "") at t1_enc.c:926
          926     in t1_enc.c
          (gdb) info locals
          hashsize = 16
          i = 72
          ctx = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0, pctx = 0x0, update = 0}
          buf = '\000' <repeats 48 times>"\320, \a HH\000\000\000`\365r\320\355*\000\000(\bK\320\355*\000\000\362x\255\070\355*\000\000`\365r\320\355*\000\000\210\365r\320\355*\000\000\250\365r\320\355*\000\000\016Q\325\070\001\000\000\000\340|i\320\355*\000\000\200\001\017\071\355*\000"
          q = 0x2aed48200750 ""
          buf2 = '\000' <repeats 11 times>
          idx = 0
          mask = 16
          err = 0
          md = 0x2aed390f7c20
          (gdb) print s->ctx
          $4 = (SSL_CTX *) 0x1344430
          (gdb) print s->s3->handshake_dgst
          $5 = (EVP_MD_CTX **) 0x2aedd06c1db0
          (gdb) print s->s3->handshake_dgst[0]
          $6 = (EVP_MD_CTX *) 0x0
          (gdb) print s->s3
          $7 = (struct ssl3_state_st *) 0x2aedd04b0810
          (gdb) print *s->s3
          $8 = {flags = 0, delay_buf_pop_ret = 0, read_sequence = "\000\000\000\000\000\000\000", read_mac_secret_size = 20, read_mac_secret = "\323b\264g7\345\362\002\222>\276\377\333\350{\204`\032\237\233", '\000' <repeats 43 times>, 
            write_sequence = "\000\000\000\000\000\000\000", write_mac_secret_size = 0, write_mac_secret = '\000' <repeats 63 times>, server_random = "R\246\063\306]\360\331\320/r\363\356S\355!\nD\021\323\n\021\035\070\302\330\300Փm\223\\\276", 
            client_random = "R\246\063\310\001\273\016͐qձL\v\341\202\235\070\216\250\262\254\343\243Q\234M,Y\001\352\063", need_empty_fragments = 0, empty_fragment_done = 0, init_extra = 0, rbuf = {buf = 0x2aee3816fbf0 "\026\003\001\024\003\001", 
              len = 16712, offset = 9, left = 0}, wbuf = {buf = 0x2aee38173d40 "P\364\061\025\003\001", len = 16560, offset = 10, left = 0}, rrec = {type = 20, length = 0, off = 0, data = 0x2aee3816fbf8 "\001", input = 0x2aee3816fbf8 "\001", comp = 0x0, 
              epoch = 0, seq_num = "\000\000\000\000\000\000\000"}, wrec = {type = 21, length = 7, off = 0, data = 0x2aee38173d48 "\002F", input = 0x2aee38173d48 "\002F", comp = 0x0, epoch = 0, seq_num = "\000\000\000\000\000\000\000"}, 
            alert_fragment = "\000", alert_fragment_len = 0, handshake_fragment = "\000\000\000", handshake_fragment_len = 0, wnum = 0, wpend_tot = 2, wpend_type = 21, wpend_ret = 2, wpend_buf = 0x2aedd04b09e8 "\002F", handshake_buffer = 0x0, 
            handshake_dgst = 0x2aedd06c1db0, change_cipher_spec = 1, warn_alert = 0, fatal_alert = 0, alert_dispatch = 0, send_alert = "\002F", renegotiate = 0, total_renegotiations = 0, num_renegotiations = 0, in_read_app_data = 0, 
            client_opaque_prf_input = 0x0, client_opaque_prf_input_len = 0, server_opaque_prf_input = 0x0, server_opaque_prf_input_len = 0, tmp = {
              cert_verify_md = "_\005X\370E\271ׄ\357\207Y\330\024\021-\216\262\303\060]\345\"\326\353ɦ\017\006'\345\a4", '\000' <repeats 95 times>, finish_md = '\000' <repeats 127 times>, finish_md_len = 0, peer_finish_md = '\000' <repeats 127 times>, 
              peer_finish_md_len = 0, message_size = 258, message_type = 16, new_cipher = 0x2aed38ce8400, dh = 0x0, ecdh = 0x0, next_state = 8576, reuse_message = 0, cert_req = 0, ctype_num = 0, ctype = "\000\000\000\000\000\000\000\000", 
              ca_names = 0x0, use_rsa_tmp = 0, key_block_length = 72, 
              key_block = 0x2aedd072f560 "\323b\264g7\345\362\002\222>\276\377\333\350{\204`\032\237\233\300\247*\350{\331c\357\221\006#_^ͭ@\366\363\247\214\067`\366ל*\323j\301\003\243=\240\031z\240\314\003\330\004\373\265\266rd\244TɗI\035\034u", 
              new_sym_enc = 0x2aed390f0180, new_hash = 0x2aed390f7d20, new_mac_pkey_type = 855, new_mac_secret_size = 20, new_compression = 0x0, cert_request = 0}, previous_client_finished = '\000' <repeats 63 times>, 
            previous_client_finished_len = 0 '\000', previous_server_finished = '\000' <repeats 63 times>, previous_server_finished_len = 0 '\000', send_connection_binding = 1, next_proto_neg_seen = 0}
          (gdb) print *md
          $9 = {type = 4, pkey_type = 8, md_size = 16, flags = 0, init = 0x2aed38e0c0bc <init>, update = 0x2aed38e0c0da <update>, final = 0x2aed38e0c10b <final>, copy = 0, cleanup = 0, sign = 0x2aed38dd8490 <RSA_sign>, 
            verify = 0x2aed38dd8ccc <RSA_verify>, required_pkey_type = {6, 19, 0, 0, 0}, block_size = 64, ctx_size = 100, md_ctrl = 0}
          (gdb) print s->s3->tmp.new_cipher->algorithm2
          $10 = 49200
          (gdb) print ctx->digest->md_size
          Cannot access memory at address 0x8
          (gdb) f 0
          #0  0x00002aed38e036b1 in EVP_DigestFinal_ex (ctx=0x2aed482007d0, md=0x2aed48200750 "", size=0x2aed48200804) at digest.c:271
          271     digest.c: No such file or directory.
                  in digest.c
          (gdb) print ctx->digest->md_size
          Cannot access memory at address 0x8
          (gdb) print *ctx
          $12 = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0, pctx = 0x0, update = 0}
          (gdb) f 3
          #3  0x00002aed38aa3c58 in ssl3_read_bytes (s=0x2aedd06d7990, type=22, buf=0x2aedd0388400 "\020", len=4, peek=0) at s3_pkt.c:1306
          1306    s3_pkt.c: No such file or directory.
                  in s3_pkt.c
          (gdb) info local
          al = 0
          i = -1116221696
          j = 247265824
          ret = 1
          n = 1679834243
          rr = 0x2aedd04b0930
          cb = 0
          (gdb) x/5b buf
          0x2aedd0388400: 0x10    0x00    0x01    0x02    0x01
          (gdb) print *rr
          $13 = {type = 20, length = 0, off = 0, data = 0x2aee3816fbf8 "\001", input = 0x2aee3816fbf8 "\001", comp = 0x0, epoch = 0, seq_num = "\000\000\000\000\000\000\000"}
          (gdb) print s->msg_callback
          $14 = (void (*)(int, int, int, const void *, size_t, SSL *, void *)) 0
          (gdb) print s->s3->tmp.new_cipher
          $15 = (const SSL_CIPHER *) 0x2aed38ce8400
          (gdb) print *s->s3->tmp.new_cipher
          $16 = {valid = 1, name = 0x2aed38ad652f "RC4-SHA", id = 50331653, algorithm_mkey = 1, algorithm_auth = 1, algorithm_enc = 4, algorithm_mac = 2, algorithm_ssl = 2, algo_strength = 65, algorithm2 = 49200, strength_bits = 128, alg_bits = 128}
          (gdb) quit
          
          Show
          Ron Barber added a comment - I have done some "debugging" on this although I know little about openssl. I have come to the conclusion that the issue lies withing openssl 1.0.1e (and 1.0.1.a) and does not exist with 1.0.0k. Specifically, I have modified TS to support "CONFIG proxy.config.ssl.TLSv1_2" which if set to 0 disables TLS 1.2 support. With TLS 1.2 disabled, TS did not crash after running over 24 hours. I crashes in less than 1 hour (during peak) with TLS 1.2 enabled. Two thoughts: 1. This ticket should be a feature request to add 2 configuration options which control enablement of TLS 1.1 and TLS 1.2 (similar to the existing TLSv1 config): "CONFIG proxy.config.ssl.TLSv1 INT 1" ** EXISTING CONFIG OPTION ** "CONFIG proxy.config.ssl.TLSv1_1 INT 1" "CONFIG proxy.config.ssl.TLSv1_2 INT 1" 2. Create a ticket/report w/openssl community for this issue. Here is a sample debug session (we are running RHEL6): Program terminated with signal 11, Segmentation fault. #0 0x00002aed38e036b1 in EVP_DigestFinal_ex (ctx=0x2aed482007d0, md=0x2aed48200750 "", size=0x2aed48200804) at digest.c:271 271 digest.c: No such file or directory. in digest.c Missing separate debuginfos, use: debuginfo-install expat-2.0.1-11.el6_2.x86_64 glibc-2.12-1.107.el6.x86_64 hwloc-1.5-1.el6.x86_64 libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64 libevent-1.4.13-4.el6.x86_64 libgcc-4.4.7-3.el6.x86_64 libstdc++-4.4.7-3.el6.x86_64 libxml2-2.7.6-12.el6_4.1.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 numactl-2.0.7-6.el6.x86_64 openssl-1.0.0-27.el6.x86_64 pciutils-libs-3.1.10-2.el6.x86_64 pcre-7.8-6.el6.x86_64 tcl-8.5.7-6.el6.x86_64 xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) where #0 0x00002aed38e036b1 in EVP_DigestFinal_ex (ctx=0x2aed482007d0, md=0x2aed48200750 "", size=0x2aed48200804) at digest.c:271 #1 0x00002aed38ab0c0b in tls1_final_finish_mac (s=0x2aedd06d7990, str=0x2aed38ad7869 "client finished", slen=15, out=0x2aedd04b0b24 "") at t1_enc.c:926 #2 0x00002aed38aa413c in ssl3_do_change_cipher_spec (s=0x2aedd06d7990) at s3_pkt.c:1462 #3 0x00002aed38aa3c58 in ssl3_read_bytes (s=0x2aedd06d7990, type=22, buf=0x2aedd0388400 "\020", len=4, peek=0) at s3_pkt.c:1306 #4 0x00002aed38aa5068 in ssl3_get_message (s=0x2aedd06d7990, st1=8608, stn=8609, mt=-1, max=516, ok=0x2aed48200a9c) at s3_both.c:451 #5 0x00002aed38a93ed7 in ssl3_get_cert_verify (s=0x2aedd06d7990) at s3_srvr.c:2924 #6 0x00002aed38a8f25c in ssl3_accept (s=0x2aedd06d7990) at s3_srvr.c:677 #7 0x00002aed38ac131c in SSL_accept (s=0x2aedd06d7990) at ssl_lib.c:940 #8 0x00000000006710ba in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aedc0129cb0, err=@0x2aed48200d1c) at SSLNetVConnection.cc:488 #9 0x0000000000672977 in SSLNetVConnection::sslStartHandShake (this=0x2aedc0129cb0, event=<value optimized out>, err=@0x2aed48200d1c) at SSLNetVConnection.cc:470 #10 0x0000000000671bd2 in SSLNetVConnection::net_read_io (this=0x2aedc0129cb0, nh=0x2aed42834bf0, lthread=0x2aed42831010) at SSLNetVConnection.cc:217 #11 0x000000000067b6b2 in NetHandler::mainNetEvent (this=0x2aed42834bf0, event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386 #12 0x00000000006a314f in handleEvent (this=0x2aed42831010, e=0x113cc70, calling_code=5) at I_Continuation.h:146 #13 EThread::process_event (this=0x2aed42831010, e=0x113cc70, calling_code=5) at UnixEThread.cc:141 #14 0x00000000006a3b33 in EThread::execute (this=0x2aed42831010) at UnixEThread.cc:265 #15 0x00000000006a1fea in spawn_thread_internal (a=0x1349630) at Thread.cc:88 #16 0x00002aed3934d851 in start_thread () from /lib64/libpthread.so.0 #17 0x000000324f0e890d in clone () from /lib64/libc.so.6 (gdb) f 7 #7 0x00002aed38ac131c in SSL_accept (s=0x2aedd06d7990) at ssl_lib.c:940 940 ssl_lib.c: No such file or directory. in ssl_lib.c (gdb) print *s $1 = {version = 769, type = 8192, method = 0x2aed38ce6e00, rbio = 0x2aedd024f760, wbio = 0x2aedd006a7e0, bbio = 0x2aedd006a7e0, rwstate = 1, in_handshake = 1, handshake_func = 0x2aed38a8e41e <ssl3_accept>, server = 1, new_session = 0, quiet_shutdown = 1, shutdown = 0, state = 8608, rstate = 240, init_buf = 0x2aedd055b2d0, init_msg = 0x2aedd0388404, init_num = 0, init_off = 0, packet = 0x2aee3816fbf3 "\024\003\001", packet_length = 0, s2 = 0x0, s3 = 0x2aedd04b0810, d1 = 0x0, read_ahead = 0, msg_callback = 0, msg_callback_arg = 0x0, hit = 0, param = 0x2aedd00060e0, cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx = 0x2aedd0697ce0, read_hash = 0x2aedd03399a0, expand = 0x0, enc_write_ctx = 0x0, write_hash = 0x0, compress = 0x0, cert = 0x2aedd00e4030, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session = 0x2aedd01cc080, generate_session_id = 0, verify_mode = 0, verify_callback = 0, info_callback = 0, error = 0, error_code = 0, psk_client_callback = 0, psk_server_callback = 0, ctx = 0x1344430, debug = 0, verify_result = 0, ex_data = {sk = 0x2aedd033a6c0, dummy = 0}, client_CA = 0x0, references = 1, options = 21102596, mode = 0, max_cert_list = 102400, first_packet = 0, client_version = 771, max_send_fragment = 16384, tlsext_debug_cb = 0, tlsext_debug_arg = 0x0, tlsext_hostname = 0x0, servername_done = 1, tlsext_status_type = -1, tlsext_status_expected = 0, tlsext_ocsp_ids = 0x0, tlsext_ocsp_exts = 0x0, tlsext_ocsp_resp = 0x0, tlsext_ocsp_resplen = -1, tlsext_ticket_expected = 1, tlsext_ecpointformatlist_length = 0, tlsext_ecpointformatlist = 0x0, tlsext_ellipticcurvelist_length = 0, tlsext_ellipticcurvelist = 0x0, tlsext_opaque_prf_input = 0x0, tlsext_opaque_prf_input_len = 0, tlsext_session_ticket = 0x0, tls_session_ticket_ext_cb = 0, tls_session_ticket_ext_cb_arg = 0x0, tls_session_secret_cb = 0, tls_session_secret_cb_arg = 0x0, initial_ctx = 0x1344430, next_proto_negotiated = 0x0, next_proto_negotiated_len = 0 '\000', srtp_profiles = 0x0, srtp_profile = 0x0, tlsext_heartbeat = 0, tlsext_hb_pending = 0, tlsext_hb_seq = 0, renegotiate = 2, srp_ctx = {SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0, SRP_verify_param_callback = 0, SRP_give_srp_client_pwd_callback = 0, login = 0x0, N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0}} (gdb) print *s->ctx $2 = {method = 0x2aed38ce6700, cipher_list = 0x1345360, cipher_list_by_id = 0x1345240, cert_store = 0x13449a0, sessions = 0x1344850, session_cache_size = 20480, session_cache_head = 0x2aede82fe200, session_cache_tail = 0x2aee080010b0, session_cache_mode = 2, session_timeout = 300, new_session_cb = 0, remove_session_cb = 0, get_session_cb = 0, stats = {sess_connect = 0, sess_connect_renegotiate = 0, sess_connect_good = 0, sess_accept = 12625, sess_accept_renegotiate = 0, sess_accept_good = 12321, sess_miss = 1549, sess_timeout = 0, sess_cache_full = 0, sess_hit = 3043, sess_cb_hit = 0}, references = 8057, app_verify_callback = 0, app_verify_arg = 0x0, default_passwd_callback = 0, default_passwd_callback_userdata = 0x0, client_cert_cb = 0, app_gen_cookie_cb = 0, app_verify_cookie_cb = 0, ex_data = {sk = 0x0, dummy = 0}, rsa_md5 = 0x2aed390f7c20, md5 = 0x2aed390f7c20, sha1 = 0x2aed390f7d20, extra_certs = 0x1346ca0, comp_methods = 0x1342ce0, info_callback = 0, client_CA = 0x1345060, options = 21102596, mode = 0, max_cert_list = 102400, cert = 0x1344720, read_ahead = 0, msg_callback = 0, msg_callback_arg = 0x0, verify_mode = 0, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, default_verify_callback = 0, generate_session_id = 0, param = 0x1345020, quiet_shutdown = 1, max_send_fragment = 16384, client_cert_engine = 0x0, tlsext_servername_callback = 0x675550 <ssl_servername_callback(SSL*, int*, void*)>, tlsext_servername_arg = 0x1343120, tlsext_tick_key_name = "\r(T[\177\025\267\216\326\213ω:\277a)", tlsext_tick_hmac_key = "]wEz9.Cȕc\237\002_--o", tlsext_tick_aes_key = "@7\333a\026cf\274\312\346\273]m\344\217A", tlsext_ticket_key_cb = 0, tlsext_status_cb = 0, tlsext_status_arg = 0x0, tlsext_opaque_prf_input_callback = 0, tlsext_opaque_prf_input_callback_arg = 0x0, psk_identity_hint = 0x0, psk_client_callback = 0, psk_server_callback = 0, freelist_max_len = 32, wbuf_freelist = 0x13451f0, rbuf_freelist = 0x13451d0, srp_ctx = {SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0, SRP_verify_param_callback = 0, SRP_give_srp_client_pwd_callback = 0, login = 0x0, N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0}, next_protos_advertised_cb = 0, next_protos_advertised_cb_arg = 0x0, next_proto_select_cb = 0, next_proto_select_cb_arg = 0x0, srtp_profiles = 0x0} (gdb) f 1 #1 0x00002aed38ab0c0b in tls1_final_finish_mac (s=0x2aedd06d7990, str=0x2aed38ad7869 "client finished", slen=15, out=0x2aedd04b0b24 "") at t1_enc.c:926 926 t1_enc.c: No such file or directory. in t1_enc.c (gdb) print *s->ctx $3 = {method = 0x2aed38ce6700, cipher_list = 0x1345360, cipher_list_by_id = 0x1345240, cert_store = 0x13449a0, sessions = 0x1344850, session_cache_size = 20480, session_cache_head = 0x2aede82fe200, session_cache_tail = 0x2aee080010b0, session_cache_mode = 2, session_timeout = 300, new_session_cb = 0, remove_session_cb = 0, get_session_cb = 0, stats = {sess_connect = 0, sess_connect_renegotiate = 0, sess_connect_good = 0, sess_accept = 12625, sess_accept_renegotiate = 0, sess_accept_good = 12321, sess_miss = 1549, sess_timeout = 0, sess_cache_full = 0, sess_hit = 3043, sess_cb_hit = 0}, references = 8057, app_verify_callback = 0, app_verify_arg = 0x0, default_passwd_callback = 0, default_passwd_callback_userdata = 0x0, client_cert_cb = 0, app_gen_cookie_cb = 0, app_verify_cookie_cb = 0, ex_data = {sk = 0x0, dummy = 0}, rsa_md5 = 0x2aed390f7c20, md5 = 0x2aed390f7c20, sha1 = 0x2aed390f7d20, extra_certs = 0x1346ca0, comp_methods = 0x1342ce0, info_callback = 0, client_CA = 0x1345060, options = 21102596, mode = 0, max_cert_list = 102400, cert = 0x1344720, read_ahead = 0, msg_callback = 0, msg_callback_arg = 0x0, verify_mode = 0, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, default_verify_callback = 0, generate_session_id = 0, param = 0x1345020, quiet_shutdown = 1, max_send_fragment = 16384, client_cert_engine = 0x0, tlsext_servername_callback = 0x675550 <ssl_servername_callback(SSL*, int*, void*)>, tlsext_servername_arg = 0x1343120, tlsext_tick_key_name = "\r(T[\177\025\267\216\326\213ω:\277a)", tlsext_tick_hmac_key = "]wEz9.Cȕc\237\002_--o", tlsext_tick_aes_key = "@7\333a\026cf\274\312\346\273]m\344\217A", tlsext_ticket_key_cb = 0, tlsext_status_cb = 0, tlsext_status_arg = 0x0, tlsext_opaque_prf_input_callback = 0, tlsext_opaque_prf_input_callback_arg = 0x0, psk_identity_hint = 0x0, psk_client_callback = 0, psk_server_callback = 0, freelist_max_len = 32, wbuf_freelist = 0x13451f0, rbuf_freelist = 0x13451d0, srp_ctx = {SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0, SRP_verify_param_callback = 0, SRP_give_srp_client_pwd_callback = 0, login = 0x0, N = 0x0, g = 0x0, s = 0x0, B = 0x0, A = 0x0, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 1024, srp_Mask = 0}, next_protos_advertised_cb = 0, next_protos_advertised_cb_arg = 0x0, next_proto_select_cb = 0, next_proto_select_cb_arg = 0x0, srtp_profiles = 0x0} (gdb) f 1 #1 0x00002aed38ab0c0b in tls1_final_finish_mac (s=0x2aedd06d7990, str=0x2aed38ad7869 "client finished", slen=15, out=0x2aedd04b0b24 "") at t1_enc.c:926 926 in t1_enc.c (gdb) info locals hashsize = 16 i = 72 ctx = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0, pctx = 0x0, update = 0} buf = '\000' <repeats 48 times>"\320, \a HH\000\000\000`\365r\320\355*\000\000(\bK\320\355*\000\000\362x\255\070\355*\000\000`\365r\320\355*\000\000\210\365r\320\355*\000\000\250\365r\320\355*\000\000\016Q\325\070\001\000\000\000\340|i\320\355*\000\000\200\001\017\071\355*\000" q = 0x2aed48200750 "" buf2 = '\000' <repeats 11 times> idx = 0 mask = 16 err = 0 md = 0x2aed390f7c20 (gdb) print s->ctx $4 = (SSL_CTX *) 0x1344430 (gdb) print s->s3->handshake_dgst $5 = (EVP_MD_CTX **) 0x2aedd06c1db0 (gdb) print s->s3->handshake_dgst[0] $6 = (EVP_MD_CTX *) 0x0 (gdb) print s->s3 $7 = (struct ssl3_state_st *) 0x2aedd04b0810 (gdb) print *s->s3 $8 = {flags = 0, delay_buf_pop_ret = 0, read_sequence = "\000\000\000\000\000\000\000", read_mac_secret_size = 20, read_mac_secret = "\323b\264g7\345\362\002\222>\276\377\333\350{\204`\032\237\233", '\000' <repeats 43 times>, write_sequence = "\000\000\000\000\000\000\000", write_mac_secret_size = 0, write_mac_secret = '\000' <repeats 63 times>, server_random = "R\246\063\306]\360\331\320/r\363\356S\355!\nD\021\323\n\021\035\070\302\330\300Փm\223\\\276", client_random = "R\246\063\310\001\273\016͐qձL\v\341\202\235\070\216\250\262\254\343\243Q\234M,Y\001\352\063", need_empty_fragments = 0, empty_fragment_done = 0, init_extra = 0, rbuf = {buf = 0x2aee3816fbf0 "\026\003\001\024\003\001", len = 16712, offset = 9, left = 0}, wbuf = {buf = 0x2aee38173d40 "P\364\061\025\003\001", len = 16560, offset = 10, left = 0}, rrec = {type = 20, length = 0, off = 0, data = 0x2aee3816fbf8 "\001", input = 0x2aee3816fbf8 "\001", comp = 0x0, epoch = 0, seq_num = "\000\000\000\000\000\000\000"}, wrec = {type = 21, length = 7, off = 0, data = 0x2aee38173d48 "\002F", input = 0x2aee38173d48 "\002F", comp = 0x0, epoch = 0, seq_num = "\000\000\000\000\000\000\000"}, alert_fragment = "\000", alert_fragment_len = 0, handshake_fragment = "\000\000\000", handshake_fragment_len = 0, wnum = 0, wpend_tot = 2, wpend_type = 21, wpend_ret = 2, wpend_buf = 0x2aedd04b09e8 "\002F", handshake_buffer = 0x0, handshake_dgst = 0x2aedd06c1db0, change_cipher_spec = 1, warn_alert = 0, fatal_alert = 0, alert_dispatch = 0, send_alert = "\002F", renegotiate = 0, total_renegotiations = 0, num_renegotiations = 0, in_read_app_data = 0, client_opaque_prf_input = 0x0, client_opaque_prf_input_len = 0, server_opaque_prf_input = 0x0, server_opaque_prf_input_len = 0, tmp = { cert_verify_md = "_\005X\370E\271ׄ\357\207Y\330\024\021-\216\262\303\060]\345\"\326\353ɦ\017\006'\345\a4", '\000' <repeats 95 times>, finish_md = '\000' <repeats 127 times>, finish_md_len = 0, peer_finish_md = '\000' <repeats 127 times>, peer_finish_md_len = 0, message_size = 258, message_type = 16, new_cipher = 0x2aed38ce8400, dh = 0x0, ecdh = 0x0, next_state = 8576, reuse_message = 0, cert_req = 0, ctype_num = 0, ctype = "\000\000\000\000\000\000\000\000", ca_names = 0x0, use_rsa_tmp = 0, key_block_length = 72, key_block = 0x2aedd072f560 "\323b\264g7\345\362\002\222>\276\377\333\350{\204`\032\237\233\300\247*\350{\331c\357\221\006#_^ͭ@\366\363\247\214\067`\366ל*\323j\301\003\243=\240\031z\240\314\003\330\004\373\265\266rd\244TɗI\035\034u", new_sym_enc = 0x2aed390f0180, new_hash = 0x2aed390f7d20, new_mac_pkey_type = 855, new_mac_secret_size = 20, new_compression = 0x0, cert_request = 0}, previous_client_finished = '\000' <repeats 63 times>, previous_client_finished_len = 0 '\000', previous_server_finished = '\000' <repeats 63 times>, previous_server_finished_len = 0 '\000', send_connection_binding = 1, next_proto_neg_seen = 0} (gdb) print *md $9 = {type = 4, pkey_type = 8, md_size = 16, flags = 0, init = 0x2aed38e0c0bc <init>, update = 0x2aed38e0c0da <update>, final = 0x2aed38e0c10b <final>, copy = 0, cleanup = 0, sign = 0x2aed38dd8490 <RSA_sign>, verify = 0x2aed38dd8ccc <RSA_verify>, required_pkey_type = {6, 19, 0, 0, 0}, block_size = 64, ctx_size = 100, md_ctrl = 0} (gdb) print s->s3->tmp.new_cipher->algorithm2 $10 = 49200 (gdb) print ctx->digest->md_size Cannot access memory at address 0x8 (gdb) f 0 #0 0x00002aed38e036b1 in EVP_DigestFinal_ex (ctx=0x2aed482007d0, md=0x2aed48200750 "", size=0x2aed48200804) at digest.c:271 271 digest.c: No such file or directory. in digest.c (gdb) print ctx->digest->md_size Cannot access memory at address 0x8 (gdb) print *ctx $12 = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0, pctx = 0x0, update = 0} (gdb) f 3 #3 0x00002aed38aa3c58 in ssl3_read_bytes (s=0x2aedd06d7990, type=22, buf=0x2aedd0388400 "\020", len=4, peek=0) at s3_pkt.c:1306 1306 s3_pkt.c: No such file or directory. in s3_pkt.c (gdb) info local al = 0 i = -1116221696 j = 247265824 ret = 1 n = 1679834243 rr = 0x2aedd04b0930 cb = 0 (gdb) x/5b buf 0x2aedd0388400: 0x10 0x00 0x01 0x02 0x01 (gdb) print *rr $13 = {type = 20, length = 0, off = 0, data = 0x2aee3816fbf8 "\001", input = 0x2aee3816fbf8 "\001", comp = 0x0, epoch = 0, seq_num = "\000\000\000\000\000\000\000"} (gdb) print s->msg_callback $14 = (void (*)(int, int, int, const void *, size_t, SSL *, void *)) 0 (gdb) print s->s3->tmp.new_cipher $15 = (const SSL_CIPHER *) 0x2aed38ce8400 (gdb) print *s->s3->tmp.new_cipher $16 = {valid = 1, name = 0x2aed38ad652f "RC4-SHA", id = 50331653, algorithm_mkey = 1, algorithm_auth = 1, algorithm_enc = 4, algorithm_mac = 2, algorithm_ssl = 2, algo_strength = 65, algorithm2 = 49200, strength_bits = 128, alg_bits = 128} (gdb) quit
          Leif Hedstrom made changes -
          Affects Version/s 4.1.2 [ 12325647 ]
          Affects Version/s 4.1.1 [ 12325577 ]
          Hide
          David Carlin added a comment -

          I upgraded from Red Hat's OpenSSL version 1.0.1e-11 RPM to the latest 1.0.1e-16 and I still see the crash.

          Show
          David Carlin added a comment - I upgraded from Red Hat's OpenSSL version 1.0.1e-11 RPM to the latest 1.0.1e-16 and I still see the crash.
          Hide
          Scott Beardsley added a comment - - edited

          Perhaps the SSLerr() call isn't being handled properly?

          Here is s3_pkt.c:

           332       /* Lets check version */
           333       if (!s->first_packet)
           334          {
           335          if (version != s->version)
           336             {
           337             SSLerr(SSL_F_SSL3_GET_RECORD,SSL_R_WRONG_VERSION_NUMBER);
           338                                 if ((s->version & 0xFF00) == (version & 0xFF00))
           339                                  /* Send back error using their minor version number :-) */
           340                s->version = (unsigned short)version;
           341             al=SSL_AD_PROTOCOL_VERSION;
           342             goto f_err;
           343             }
           344          }
           345 
          

          ....

          1250       else if (alert_level == 2) /* fatal */
          1251          {
          1252          char tmp[16];
          1253 
          1254          s->rwstate=SSL_NOTHING;
          1255          s->s3->fatal_alert = alert_descr;
          1256          SSLerr(SSL_F_SSL3_READ_BYTES, SSL_AD_REASON_OFFSET + alert_descr);
          1257          BIO_snprintf(tmp,sizeof tmp,"%d",alert_descr);
          1258          ERR_add_error_data(2,"SSL alert number ",tmp);
          1259          s->shutdown|=SSL_RECEIVED_SHUTDOWN;
          1260          SSL_CTX_remove_session(s->ctx,s->session);
          1261          return(0);
          1262          }
          
          Show
          Scott Beardsley added a comment - - edited Perhaps the SSLerr() call isn't being handled properly? Here is s3_pkt.c: 332 /* Lets check version */ 333 if (!s->first_packet) 334 { 335 if (version != s->version) 336 { 337 SSLerr(SSL_F_SSL3_GET_RECORD,SSL_R_WRONG_VERSION_NUMBER); 338 if ((s->version & 0xFF00) == (version & 0xFF00)) 339 /* Send back error using their minor version number :-) */ 340 s->version = (unsigned short)version; 341 al=SSL_AD_PROTOCOL_VERSION; 342 goto f_err; 343 } 344 } 345 .... 1250 else if (alert_level == 2) /* fatal */ 1251 { 1252 char tmp[16]; 1253 1254 s->rwstate=SSL_NOTHING; 1255 s->s3->fatal_alert = alert_descr; 1256 SSLerr(SSL_F_SSL3_READ_BYTES, SSL_AD_REASON_OFFSET + alert_descr); 1257 BIO_snprintf(tmp,sizeof tmp,"%d",alert_descr); 1258 ERR_add_error_data(2,"SSL alert number ",tmp); 1259 s->shutdown|=SSL_RECEIVED_SHUTDOWN; 1260 SSL_CTX_remove_session(s->ctx,s->session); 1261 return(0); 1262 }
          David Carlin made changes -
          Affects Version/s 4.0.1 [ 12324656 ]
          Affects Version/s 4.1.1 [ 12325577 ]
          Leif Hedstrom made changes -
          Fix Version/s 4.2.0 [ 12324892 ]
          Igor Galić made changes -
          Description I upgraded some 4.0.1 and 4.0.2 hosts from OpenSSL 1.0.0 to 1.0.1e which is supposed to be ABI compatible. I see this crash about 10 times in a given 24 hour period.

          I'm interested in OpenSSL 1.0.1e as there is a CPU usage improvement in my tests, and for TLS 1.2 support.

          I came across this squid bug with a very similar backtrace. The OpenSSL RT ticket says

          "I have discussed this situation with some Squid developers and we decided - after SSL error 1408F10B calling standard/raw read() instead of SSL_read() for empty socket buffer and this patch stopped crash Squid."

          http://rt.openssl.org/Ticket/Display.html?id=3128&user=guest&pass=guest

          {noformat}
          #0 0x0000003f842e7154 in EVP_DigestFinal_ex () from /usr/lib64/libcrypto.so.10
          #1 0x0000003f84636263 in tls1_final_finish_mac () from /usr/lib64/libssl.so.10
          #2 0x0000003f8462ad62 in ssl3_do_change_cipher_spec () from /usr/lib64/libssl.so.10
          #3 0x0000003f8462c7f7 in ssl3_read_bytes () from /usr/lib64/libssl.so.10
          #4 0x0000003f8462d5e2 in ssl3_get_message () from /usr/lib64/libssl.so.10
          #5 0x0000003f8461da1c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10
          #6 0x0000003f84621e78 in ssl3_accept () from /usr/lib64/libssl.so.10
          #7 0x00000000006711aa in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aadd0024300,
              err=@0x2aacab940c5c) at SSLNetVConnection.cc:488
          #8 0x0000000000672b77 in SSLNetVConnection::sslStartHandShake (this=0x2aadd0024300,
              event=<value optimized out>, err=@0x2aacab940c5c) at SSLNetVConnection.cc:470
          #9 0x0000000000671dd2 in SSLNetVConnection::net_read_io (this=0x2aadd0024300, nh=
              0x2aacaa02cbf0, lthread=0x2aacaa029010) at SSLNetVConnection.cc:217
          #10 0x000000000067b8c2 in NetHandler::mainNetEvent (this=0x2aacaa02cbf0,
              event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
          #11 0x00000000006a335f in handleEvent (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at I_Continuation.h:146
          #12 EThread::process_event (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at UnixEThread.cc:141
          #13 0x00000000006a3d43 in EThread::execute (this=0x2aacaa029010) at UnixEThread.cc:265
          #14 0x00000000006a21fa in spawn_thread_internal (a=0x143ec30) at Thread.cc:88
          #15 0x00002aaca05b9851 in start_thread () from /lib64/libpthread.so.0
          #16 0x000000324f0e890d in clone () from /lib64/libc.so.6
          {noformat}

          {noformat}
          NOTE: Traffic Server received Sig 11: Segmentation fault
          /home/y/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0x324f40f500)[0x2b523d64e500]
          /usr/lib64/libcrypto.so.10(EVP_DigestFinal_ex+0x24)[0x3f842e7154]
          /usr/lib64/libssl.so.10(tls1_final_finish_mac+0x233)[0x3f84636263]
          /usr/lib64/libssl.so.10(ssl3_do_change_cipher_spec+0x72)[0x3f8462ad62]
          /usr/lib64/libssl.so.10(ssl3_read_bytes+0xb57)[0x3f8462c7f7]
          /usr/lib64/libssl.so.10(ssl3_get_message+0x222)[0x3f8462d5e2]
          /usr/lib64/libssl.so.10(ssl3_get_cert_verify+0x6c)[0x3f8461da1c]
          /usr/lib64/libssl.so.10(ssl3_accept+0x788)[0x3f84621e78]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x2a)[0x6711aa]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x37)[0x672b77]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x1f2)[0x671dd2]
          /home/y/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x67b8c2]
          /home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a335f]
          /home/y/bin/traffic_server(_ZN7EThread7executeEv+0x4a3)[0x6a3d43]
          /home/y/bin/traffic_server[0x6a21fa]
          /lib64/libpthread.so.0(+0x324f407851)[0x2b523d646851]
          /lib64/libc.so.6(clone+0x6d)[0x324f0e890d]
          {noformat}
          I upgraded some 4.0.1 and 4.0.2 hosts from OpenSSL 1.0.0 to 1.0.1e which is supposed to be ABI compatible. I see this crash about 10 times in a given 24 hour period.

          I'm interested in OpenSSL 1.0.1e as there is a CPU usage improvement in my tests, and for TLS 1.2 support.

          I came across this squid bug with a very similar backtrace. The OpenSSL RT ticket says

          "I have discussed this situation with some Squid developers and we decided - after SSL error 1408F10B calling standard/raw read() instead of SSL_read() for empty socket buffer and this patch stopped crash Squid."

          http://rt.openssl.org/Ticket/Display.html?id=3128&user=guest&pass=guest

          {noformat}
          #0 0x0000003f842e7154 in EVP_DigestFinal_ex () from /usr/lib64/libcrypto.so.10
          #1 0x0000003f84636263 in tls1_final_finish_mac () from /usr/lib64/libssl.so.10
          #2 0x0000003f8462ad62 in ssl3_do_change_cipher_spec () from /usr/lib64/libssl.so.10
          #3 0x0000003f8462c7f7 in ssl3_read_bytes () from /usr/lib64/libssl.so.10
          #4 0x0000003f8462d5e2 in ssl3_get_message () from /usr/lib64/libssl.so.10
          #5 0x0000003f8461da1c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10
          #6 0x0000003f84621e78 in ssl3_accept () from /usr/lib64/libssl.so.10
          #7 0x00000000006711aa in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aadd0024300,
              err=@0x2aacab940c5c) at SSLNetVConnection.cc:488
          #8 0x0000000000672b77 in SSLNetVConnection::sslStartHandShake (this=0x2aadd0024300,
              event=<value optimized out>, err=@0x2aacab940c5c) at SSLNetVConnection.cc:470
          #9 0x0000000000671dd2 in SSLNetVConnection::net_read_io (this=0x2aadd0024300, nh=
              0x2aacaa02cbf0, lthread=0x2aacaa029010) at SSLNetVConnection.cc:217
          #10 0x000000000067b8c2 in NetHandler::mainNetEvent (this=0x2aacaa02cbf0,
              event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
          #11 0x00000000006a335f in handleEvent (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at I_Continuation.h:146
          #12 EThread::process_event (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at UnixEThread.cc:141
          #13 0x00000000006a3d43 in EThread::execute (this=0x2aacaa029010) at UnixEThread.cc:265
          #14 0x00000000006a21fa in spawn_thread_internal (a=0x143ec30) at Thread.cc:88
          #15 0x00002aaca05b9851 in start_thread () from /lib64/libpthread.so.0
          #16 0x000000324f0e890d in clone () from /lib64/libc.so.6
          {noformat}

          {noformat}
          NOTE: Traffic Server received Sig 11: Segmentation fault
          /home/y/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0x324f40f500)[0x2b523d64e500]
          /usr/lib64/libcrypto.so.10(EVP_DigestFinal_ex+0x24)[0x3f842e7154]
          /usr/lib64/libssl.so.10(tls1_final_finish_mac+0x233)[0x3f84636263]
          /usr/lib64/libssl.so.10(ssl3_do_change_cipher_spec+0x72)[0x3f8462ad62]
          /usr/lib64/libssl.so.10(ssl3_read_bytes+0xb57)[0x3f8462c7f7]
          /usr/lib64/libssl.so.10(ssl3_get_message+0x222)[0x3f8462d5e2]
          /usr/lib64/libssl.so.10(ssl3_get_cert_verify+0x6c)[0x3f8461da1c]
          /usr/lib64/libssl.so.10(ssl3_accept+0x788)[0x3f84621e78]
          /home/y/bin/traffic_server(SSLNetVConnection::sslServerHandShakeEvent(int&)+0x2a)[0x6711aa]
          /home/y/bin/traffic_server(SSLNetVConnection::sslStartHandShake(int, int&)+0x37)[0x672b77]
          /home/y/bin/traffic_server(SSLNetVConnection::net_read_io(NetHandler*, EThread*)+0x1f2)[0x671dd2]
          /home/y/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1f2)[0x67b8c2]
          /home/y/bin/traffic_server(EThread::process_event(Event*, int)+0x8f)[0x6a335f]
          /home/y/bin/traffic_server(EThread::execute()+0x4a3)[0x6a3d43]
          /home/y/bin/traffic_server[0x6a21fa]
          /lib64/libpthread.so.0(+0x324f407851)[0x2b523d646851]
          /lib64/libc.so.6(clone+0x6d)[0x324f0e890d]
          {noformat}
          Hide
          David Carlin added a comment - - edited

          I don't see error 1408F10B before the crash like how is reported for squid. Host crashed at 21:44 and the previous instance of 1408F10B is 2.5 hours earlier. FYI - At this time (19:11) the host was running a build of ATS built against OpenSSL 1.0.0. In between that time and the crash whose logs appear below at 21:44, I had replaced ATS on the host with one built against OpenSSL 1.0.1e per suggestions on IRC.

          [Nov 15 19:11:15.871] Server {0x2b5f1a931700} ERROR: SSL::25:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:337:
          

          SSL errors just before the crash:

          [Nov 15 21:44:03.572] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:03.955] Server {0x2b524da19700} ERROR: SSL::45:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1256:SSL alert number 42
          [Nov 15 21:44:04.313] Server {0x2b524dc1b700} ERROR: SSL::47:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:08.201] Server {0x2b5244100700} NOTE: Traffic Server is skipping the current log entry for squid.log because its size (87144) exceeds the maximum payload space in a log buffer
          [Nov 15 21:44:12.798] Server {0x2b5244e0d700} NOTE: Traffic Server is skipping the current log entry for squid.log because its size (16528) exceeds the maximum payload space in a log buffer
          [Nov 15 21:44:14.101] Server {0x2b524cd0c700} ERROR: SSL::32:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:16.352] Server {0x2b524c504700} ERROR: SSL::24:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:16.600] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:25.645] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:27.051] Server {0x2b524cc0b700} ERROR: SSL::31:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:29.943] Server {0x2b524ce0d700} ERROR: SSL::33:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:31.863] Server {0x2b524cb0a700} ERROR: SSL::30:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:32.284] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:33.168] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
          [Nov 15 21:44:38.135] {0x2aaca1066640} STATUS: opened /home/y/logs/trafficserver/diags.log
          
          Show
          David Carlin added a comment - - edited I don't see error 1408F10B before the crash like how is reported for squid. Host crashed at 21:44 and the previous instance of 1408F10B is 2.5 hours earlier. FYI - At this time (19:11) the host was running a build of ATS built against OpenSSL 1.0.0. In between that time and the crash whose logs appear below at 21:44, I had replaced ATS on the host with one built against OpenSSL 1.0.1e per suggestions on IRC. [Nov 15 19:11:15.871] Server {0x2b5f1a931700} ERROR: SSL::25:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:337: SSL errors just before the crash: [Nov 15 21:44:03.572] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:03.955] Server {0x2b524da19700} ERROR: SSL::45:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1256:SSL alert number 42 [Nov 15 21:44:04.313] Server {0x2b524dc1b700} ERROR: SSL::47:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:08.201] Server {0x2b5244100700} NOTE: Traffic Server is skipping the current log entry for squid.log because its size (87144) exceeds the maximum payload space in a log buffer [Nov 15 21:44:12.798] Server {0x2b5244e0d700} NOTE: Traffic Server is skipping the current log entry for squid.log because its size (16528) exceeds the maximum payload space in a log buffer [Nov 15 21:44:14.101] Server {0x2b524cd0c700} ERROR: SSL::32:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:16.352] Server {0x2b524c504700} ERROR: SSL::24:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:16.600] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:25.645] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:27.051] Server {0x2b524cc0b700} ERROR: SSL::31:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:29.943] Server {0x2b524ce0d700} ERROR: SSL::33:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:31.863] Server {0x2b524cb0a700} ERROR: SSL::30:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:32.284] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:33.168] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0 [Nov 15 21:44:38.135] {0x2aaca1066640} STATUS: opened /home/y/logs/trafficserver/diags.log
          David Carlin made changes -
          Description I upgraded some 4.0.1 and 4.0.2 hosts from OpenSSL 1.0.0 to 1.0.1e which is supposed to be ABI compatible. I see this crash about 10 times in a given 24 hour period.

          I'm interested in OpenSSL 1.0.1e as there is a CPU usage improvement in my tests, and for TLS 1.2 support.

          I came across this squid bug with a very similar backtrace. The RT ticket says

          "I have discussed this situation with some Squid developers and we decided - after SSL error 1408F10B calling standard/raw read() instead of SSL_read() for empty socket buffer and this patch stopped crash Squid."

          http://rt.openssl.org/Ticket/Display.html?id=3128&user=guest&pass=guest

          {noformat}
          #0 0x0000003f842e7154 in EVP_DigestFinal_ex () from /usr/lib64/libcrypto.so.10
          #1 0x0000003f84636263 in tls1_final_finish_mac () from /usr/lib64/libssl.so.10
          #2 0x0000003f8462ad62 in ssl3_do_change_cipher_spec () from /usr/lib64/libssl.so.10
          #3 0x0000003f8462c7f7 in ssl3_read_bytes () from /usr/lib64/libssl.so.10
          #4 0x0000003f8462d5e2 in ssl3_get_message () from /usr/lib64/libssl.so.10
          #5 0x0000003f8461da1c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10
          #6 0x0000003f84621e78 in ssl3_accept () from /usr/lib64/libssl.so.10
          #7 0x00000000006711aa in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aadd0024300,
              err=@0x2aacab940c5c) at SSLNetVConnection.cc:488
          #8 0x0000000000672b77 in SSLNetVConnection::sslStartHandShake (this=0x2aadd0024300,
              event=<value optimized out>, err=@0x2aacab940c5c) at SSLNetVConnection.cc:470
          #9 0x0000000000671dd2 in SSLNetVConnection::net_read_io (this=0x2aadd0024300, nh=
              0x2aacaa02cbf0, lthread=0x2aacaa029010) at SSLNetVConnection.cc:217
          #10 0x000000000067b8c2 in NetHandler::mainNetEvent (this=0x2aacaa02cbf0,
              event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
          #11 0x00000000006a335f in handleEvent (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at I_Continuation.h:146
          #12 EThread::process_event (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at UnixEThread.cc:141
          #13 0x00000000006a3d43 in EThread::execute (this=0x2aacaa029010) at UnixEThread.cc:265
          #14 0x00000000006a21fa in spawn_thread_internal (a=0x143ec30) at Thread.cc:88
          #15 0x00002aaca05b9851 in start_thread () from /lib64/libpthread.so.0
          #16 0x000000324f0e890d in clone () from /lib64/libc.so.6
          {noformat}

          {noformat}
          NOTE: Traffic Server received Sig 11: Segmentation fault
          /home/y/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0x324f40f500)[0x2b523d64e500]
          /usr/lib64/libcrypto.so.10(EVP_DigestFinal_ex+0x24)[0x3f842e7154]
          /usr/lib64/libssl.so.10(tls1_final_finish_mac+0x233)[0x3f84636263]
          /usr/lib64/libssl.so.10(ssl3_do_change_cipher_spec+0x72)[0x3f8462ad62]
          /usr/lib64/libssl.so.10(ssl3_read_bytes+0xb57)[0x3f8462c7f7]
          /usr/lib64/libssl.so.10(ssl3_get_message+0x222)[0x3f8462d5e2]
          /usr/lib64/libssl.so.10(ssl3_get_cert_verify+0x6c)[0x3f8461da1c]
          /usr/lib64/libssl.so.10(ssl3_accept+0x788)[0x3f84621e78]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x2a)[0x6711aa]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x37)[0x672b77]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x1f2)[0x671dd2]
          /home/y/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x67b8c2]
          /home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a335f]
          /home/y/bin/traffic_server(_ZN7EThread7executeEv+0x4a3)[0x6a3d43]
          /home/y/bin/traffic_server[0x6a21fa]
          /lib64/libpthread.so.0(+0x324f407851)[0x2b523d646851]
          /lib64/libc.so.6(clone+0x6d)[0x324f0e890d]
          {noformat}
          I upgraded some 4.0.1 and 4.0.2 hosts from OpenSSL 1.0.0 to 1.0.1e which is supposed to be ABI compatible. I see this crash about 10 times in a given 24 hour period.

          I'm interested in OpenSSL 1.0.1e as there is a CPU usage improvement in my tests, and for TLS 1.2 support.

          I came across this squid bug with a very similar backtrace. The OpenSSL RT ticket says

          "I have discussed this situation with some Squid developers and we decided - after SSL error 1408F10B calling standard/raw read() instead of SSL_read() for empty socket buffer and this patch stopped crash Squid."

          http://rt.openssl.org/Ticket/Display.html?id=3128&user=guest&pass=guest

          {noformat}
          #0 0x0000003f842e7154 in EVP_DigestFinal_ex () from /usr/lib64/libcrypto.so.10
          #1 0x0000003f84636263 in tls1_final_finish_mac () from /usr/lib64/libssl.so.10
          #2 0x0000003f8462ad62 in ssl3_do_change_cipher_spec () from /usr/lib64/libssl.so.10
          #3 0x0000003f8462c7f7 in ssl3_read_bytes () from /usr/lib64/libssl.so.10
          #4 0x0000003f8462d5e2 in ssl3_get_message () from /usr/lib64/libssl.so.10
          #5 0x0000003f8461da1c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10
          #6 0x0000003f84621e78 in ssl3_accept () from /usr/lib64/libssl.so.10
          #7 0x00000000006711aa in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aadd0024300,
              err=@0x2aacab940c5c) at SSLNetVConnection.cc:488
          #8 0x0000000000672b77 in SSLNetVConnection::sslStartHandShake (this=0x2aadd0024300,
              event=<value optimized out>, err=@0x2aacab940c5c) at SSLNetVConnection.cc:470
          #9 0x0000000000671dd2 in SSLNetVConnection::net_read_io (this=0x2aadd0024300, nh=
              0x2aacaa02cbf0, lthread=0x2aacaa029010) at SSLNetVConnection.cc:217
          #10 0x000000000067b8c2 in NetHandler::mainNetEvent (this=0x2aacaa02cbf0,
              event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
          #11 0x00000000006a335f in handleEvent (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at I_Continuation.h:146
          #12 EThread::process_event (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
              at UnixEThread.cc:141
          #13 0x00000000006a3d43 in EThread::execute (this=0x2aacaa029010) at UnixEThread.cc:265
          #14 0x00000000006a21fa in spawn_thread_internal (a=0x143ec30) at Thread.cc:88
          #15 0x00002aaca05b9851 in start_thread () from /lib64/libpthread.so.0
          #16 0x000000324f0e890d in clone () from /lib64/libc.so.6
          {noformat}

          {noformat}
          NOTE: Traffic Server received Sig 11: Segmentation fault
          /home/y/bin/traffic_server - STACK TRACE:
          /lib64/libpthread.so.0(+0x324f40f500)[0x2b523d64e500]
          /usr/lib64/libcrypto.so.10(EVP_DigestFinal_ex+0x24)[0x3f842e7154]
          /usr/lib64/libssl.so.10(tls1_final_finish_mac+0x233)[0x3f84636263]
          /usr/lib64/libssl.so.10(ssl3_do_change_cipher_spec+0x72)[0x3f8462ad62]
          /usr/lib64/libssl.so.10(ssl3_read_bytes+0xb57)[0x3f8462c7f7]
          /usr/lib64/libssl.so.10(ssl3_get_message+0x222)[0x3f8462d5e2]
          /usr/lib64/libssl.so.10(ssl3_get_cert_verify+0x6c)[0x3f8461da1c]
          /usr/lib64/libssl.so.10(ssl3_accept+0x788)[0x3f84621e78]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x2a)[0x6711aa]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x37)[0x672b77]
          /home/y/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x1f2)[0x671dd2]
          /home/y/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x67b8c2]
          /home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a335f]
          /home/y/bin/traffic_server(_ZN7EThread7executeEv+0x4a3)[0x6a3d43]
          /home/y/bin/traffic_server[0x6a21fa]
          /lib64/libpthread.so.0(+0x324f407851)[0x2b523d646851]
          /lib64/libc.so.6(clone+0x6d)[0x324f0e890d]
          {noformat}
          David Carlin made changes -
          Field Original Value New Value
          Summary ATS 4.x crashes when using OpenSSL 1.0.1e ATS 4.0.x crashes when using OpenSSL 1.0.1e
          Hide
          David Carlin added a comment -

          Forgot to mention.. igalic and postwait suggested upgrading my build environment to openssl 1.0.1e.. I tried this, didn't help.

          Show
          David Carlin added a comment - Forgot to mention.. igalic and postwait suggested upgrading my build environment to openssl 1.0.1e.. I tried this, didn't help.
          David Carlin created issue -

            People

            • Assignee:
              Bryan Call
              Reporter:
              David Carlin
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development