[SERF-198] OpenSSL BIO control method incorrectly handles unknown requests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: serf-1.3.9, serf-trunk
• Fix Version/s: serf-1.3.10
• Environment: FreeBSD 14 with KTLS enabled-OpenSSL and the base system svnlite using a bundled serf. Has also been observed with subversion + serf built from FreeBSD ports.
• Flags: Patch
• Assignee: Unassigned
• Reporter: John Baldwin
• Created: 03/Feb/21 19:22
• Updated: 31/May/23 19:52
• Resolved: 31/May/23 19:52

Description

According to the BIO_ctrl(3) manpage from OpenSSL, control methods in custom BIO classes should return 0 for unknown control requests:

Source/sink BIOs return an 0 if they do not recognize the BIO_ctrl() operation.

ssl_buckets.c includes two custom BIO classes both of which are sink BIOs, but the custom control method returns 1 instead of 0 for unknown operations.  This causes breakage with newer version of OpenSSL.  In particular, in OpenSSL versions supporting KTLS, this causes OpenSSL to believe that the custom BIOs support KTLS and thus handle TLS header insertion and encryption/decryption in the BIO layer breaking the use of HTTPS.  This was observed in FreeBSD when FreeBSD integrated KTLS support into OpenSSL:

253135

The patch below changes the default value of the control methods to 0 which fixes the KTLS case.