Summary: | Memory leak when performing client certificate validation with OCSP | ||
---|---|---|---|
Product: | Tomcat Native | Reporter: | Sander Benschop <sander.benschop> |
Component: | Library | Assignee: | Tomcat Developers Mailing List <dev> |
Status: | RESOLVED FIXED | ||
Severity: | critical | ||
Priority: | P2 | ||
Version: | 1.2.17 | ||
Target Milestone: | --- | ||
Hardware: | PC | ||
OS: | Linux | ||
Attachments: | Figures 1 & 2 |
Description
Sander Benschop
2018-11-07 15:15:05 UTC
I have further isolated the issue by replacing the verify_cb function 'SSL_callback_SSL_verify' (from the Tomcat Native Library) with a no-op function. When I do this the available memory remains constant, our test server didn't run out of memory all weekend with the same polling frequency as before. replacing SSL_callback_SSL_verify() by no-op disable all the OSCP checks, that is probably not what you want to do... But yes that shows that the leak is somewhere in SSL_callback_SSL_verify(). You are correct jfclere, I indeed only tried this in an attempt to isolate the cause of the leak. I should have been more clear in my previous comment :-) The problem is OCSP_parse_url() we have forgotten: OPENSSL_free(hostname); OPENSSL_free(c_port); OPENSSL_free(path); I will commit the fix tomorrow, testing it now. Try with r1846499, I still have another memory leak but can't find where. I was getting errors in the Python build script when running the buildconf file: ImportError: No module named 'ConfigParser' And I tried to run the buildcheck.sh file which reported I didn't have Python installed, but I do: sander:/tmp$ python Python 2.7.12 (default, Dec 4 2017, 14:50:18) So for now I've applied the patch you suggested to the downloaded sources of Tomcat Native Library 1.2.17 I was using. Thank you for the fix! I will report back in a few hours. ImportError: No module named 'ConfigParser' that is because you are using python... You need an apr version that supports python3 or use python2. Ok, I will try again to build the code from SVN and see if it makes a difference, but right now the server still runs out of memory. I have added the three lines of code you suggested in this place: free_bio: BIO_free(bio_req); free_req: if(apr_sock && ok) /* if ok == 0 we have already closed the socket */ apr_socket_close(apr_sock); apr_pool_destroy(mp); sk_OCSP_CERTID_free(ids); OCSP_REQUEST_free(ocsp_req); // Manually added code OPENSSL_free(hostname); OPENSSL_free(c_port); OPENSSL_free(path); // End manually added code end: return ocsp_resp; It seems that this does have a positive effect on the memory usage, it now took 4,5 hours to run out of memory rather than 3 but the end result is still the same. I will report back when I've tried the exact commit in SVN. OK I know that adding: OPENSSL_free(hostname); OPENSSL_free(c_port); OPENSSL_free(path); is not enough, but I am happy it helps ;-) try with http://svn.apache.org/viewvc?rev=1846593&view=rev I think I have fixed all the leaks now. It worked! I updated the code yesterday and the server still hasn't run out of memory. In the graph I can see that it stabilises nicely. Thank you so much jfclere! :-) |