[HBASE-27185] Rewrite NettyRpcServer to decode rpc request with netty handler

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-alpha-4
• Component/s: netty, rpc
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 08/Jul/22 14:20
• Updated: 29/Jul/22 19:48
• Resolved: 28/Jul/22 09:38

Description

This is a follow on issue of HBASE-26708.

The memory leaks happens in the saslReadAndProcess method. It does not follow a typical netty style so it is more likely to introduce bugs.

Since we agree that NettyRpcServer is the future and SimpleRpcServer should be deprecated and finally removed, let's first implement dedicated rpc request decoder for netty.

Issue Links

• is related to: HBASE-26708 Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering with SASL implementation
• links to: GitHub Pull Request #4624