[KAFKA-7454] Use lazy allocation for SslTransportLayer buffers

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0.3, 1.0.2, 1.1.1, 2.0.0
• Fix Version/s: 1.1.2, 2.0.1, 2.1.0
• Component/s: security
• Assignee: Rajini Sivaram
• Reporter: Rajini Sivaram
• Created: 28/Sep/18 14:47
• Updated: 29/Sep/18 09:20
• Resolved: 29/Sep/18 08:20

Description

At the moment, three heap buffers are allocated for SslTransportLayers at the time when the instance is created (before establishing the connection on the client-side and when accepting the connection on the broker-side). When there are a large number of connections and the broker is overloaded, this can result in unnecessary memory pressure on the broker due to client reconnections since buffers may be allocated unnecessarily for client connections whose handshake is never processed. It will be better to lazily allocate buffers to reduce memory pressure. On the broker-side, buffers will be allocated when the first packet is received from the client, starting handshake. On the client-side, buffers will be allocated once the connection is established when the client initiates handshake.

Issue Links

• relates to: KAFKA-7304 memory leakage in org.apache.kafka.common.network.Selector
• links to: GitHub Pull Request #5713