There is a block of code in checkHandshake that attempts to set read/write memory bios to be nonblocking. This code doesn't do anything:
Here's what this code looks like, and the problems:
- BIO_new(BIO_s_mem()) creates a new memory BIO. Not sure why.
- BIO_set_nbio() executes BIO_ctrl(..., BIO_C_SET_NBIO, ...). This errors out and return 0 because mem_ctrl does not have a case for BIO_C_SET_NBIO. See: https://github.com/openssl/openssl/blob/6f0ac0e2f27d9240516edb9a23b7863e7ad02898/crypto/bio/bss_mem.c#L226
- SSL_set_bio() sets the SSL* to use the memory BIOs.
- SSL_set_fd() creates a socket BIO, sets the FD on it, and uses SSL_set_bio() to replace the memory BIOs.
As far as I can tell, this block of code does nothing and will not change functionality. If there's a reason that it's there, it needs to be re-implemented.