[PROTON-2736] TLS OpenSSL library: hang with large application data frames

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: proton-c-0.38.0
• Fix Version/s: proton-c-0.39.0
• Component/s: proton-c
• Assignee: Clifford Jansen
• Reporter: Clifford Jansen
• Created: 14/May/23 18:28
• Updated: 07/Jun/23 12:18
• Resolved: 29/May/23 16:59

Description

OpenSSL maintains a buffer large enough for the largest possible TLS protocol record + 1K.  The Proton TLS decrypt loop is unaware of record boundaries and repeatedly adds encrypted bytes at one end and takes out decrypted bytes at the other, stopping when there is no more to decrypt or no more application buffer space to move decrypted content into.

It also tests if there are remaining decrypted bytes available should the application provide additional buffers.  This test can fail in the case that the OpenSSL buffer is completely filled with:

 handshake record > 1K followed by
 partial max sized application data record

The SSL_peek operation will not see any application data and Proton "remembers" the full buffer without allowing that the handshake record has been processed and the buffer is no longer full.