[KAFKA-15502] Handle large keystores in SslEngineValidator

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1, 3.6.0, 3.5.1
• Fix Version/s: 3.4.2, 3.5.2, 3.7.0, 3.6.1
• Assignee: Manikumar
• Reporter: Manikumar
• Created: 25/Sep/23 15:56
• Updated: 08/Oct/23 07:03
• Resolved: 08/Oct/23 07:03

Description

We have observed an issue where inter broker SSL listener is not coming up for large keystores (size >16K)

1. Currently validator code doesn't work well with large stores. Right now, WRAP returns if there is already data in the buffer. But if we need more data to be wrapped for UNWRAP to succeed, we end up looping forever.

2. Observed large TLSv3 post handshake messages are not getting read and causing validator code loop forever. This is observed with JDK17+
 

Issue Links

• links to: GitHub Pull Request #14445