[THRIFT-5482] Memory leak during SSL handshake

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.16.0
• Component/s: C++ - Library

Environment

Build env: Windows 2019

Issue observed: Windows 2012R2

OpenSSL: 1.1.1k

• Assignee: Anshul Mohan Gupta
• Reporter: Anshul Mohan Gupta
• Created: 01/Dec/21 08:37
• Updated: 30/Aug/22 21:31
• Resolved: 18/May/22 01:33

Description

We observed continuous memory increase in the thrift server during SSL handshake and this memory increase was crashing the windows hosts as there was no memory left for other applications to work. During further investigation, we found that ERR_remove_state has been made stub in the later versions of OpenSSL, and recommended function to clear the thread state is OPENSSL_thread_stop. Once I made this change we did not observe any memory leak further. We are using OpenSSL 1.1.1k with thrift 0.12.0. Hence I am putting the patch for the same. Thanks!

I am adding call stack sample leak trace

Call stack sample 2
Address   0x0000000e`00bdd770
Allocation Time  00:00:45 since tracking started
Allocation Size 64 Bytes


Function
LeakTrack+1f109
!_malloc_base+44
!CRYPTO_zalloc+59
!BN_CTX_new+20
!rsa_ossl_public_decrypt+d8
!pkey_rsa_verify+245
!EVP_DigestVerify+15f
!tls_process_cert_verify+42a
!read_state_machine+24f
!state_machine+2d2
!SSL_do_handshake+bc
!apache::thrift::transport::TSSLSocket::initializeHandshake+dc
!apache::thrift::transport::TSSLSocket::read+3c
!apache::thrift::transport::TBufferedTransport::readSlow+56
!apache::thrift::transport::TBufferBase::readAll+88
!apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport,apache::thrift::protocol::TNetworkBigEndian>,apache::thrift::protocol::TProtocolDefaults>::readMessageBegin_virt+4c
!apache::thrift::TMultiplexedProcessor::process+87
!apache::thrift::server::TConnectedClient::run+16a
!apache::thrift::server::TThreadedServer::TConnectedClientRunner::run+17
!apache::thrift::concurrency::StdThread::threadMain+6c
!std::thread::_Invoke<std::tuple<int (__cdecl*)(std::shared_ptr<Reducted>),std::shared_ptr<Reducted> >,0,1>+3e
!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+50
kernel32!BaseThreadInitThunk+22
ntdll!RtlUserThreadStart+34

Issue Links

• links to: GitHub Pull Request #2482