[HBASE-28010] Connection attributes can become corrupted on the server side

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-beta-1
• Assignee: Ray Mattingly
• Reporter: Bryan Beaudreault
• Created: 05/Aug/23 01:13
• Updated: 24/Aug/23 19:59
• Resolved: 23/Aug/23 18:22

Description

In ServerRpcConnection.processOneRpc, it calls processConnectionHeader and then immediately calls callCleanupIfNeeded. The parsing of the ByteBuff into the ConnectionHeader does not copy the bytes. We keep a reference to ConnectionHeader for later use, but since the underlying ByteBuff gets released in callCleanupIfNeeded, later requests can override the memory locations that the ConnectionHeader points at.

The unit tests we added dont catch this possibly because they don't send enough requests to corrupt the buffers. It happens pretty quickly in a deployed cluster.

We need to copy the List<NameBytesPair> from the ConnectionHeader into a Map before the buffer is released. This probably means we should remove getConnectionHeader from the RpcCall interface and instead add getConnectionAttributes.

Issue Links

• links to: GitHub Pull Request #5366