[ZOOKEEPER-4748] quorum.QuorumCnxManager: BufferUnderflowException

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: quorum
• Assignee: Unassigned
• Reporter: Ke Han
• Created: 25/Sep/23 19:45
• Updated: 04/Oct/23 00:38

Description

When running zookeeper (3.5.7, integrated in HBase-2.4.7), I met the following error message.

2023-09-25T11:24:41,326 ERROR [SendWorker:1] quorum.QuorumCnxManager: BufferUnderflowException
java.nio.BufferUnderflowException: null
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:155) ~[?:1.8.0_362]
        at java.nio.ByteBuffer.get(ByteBuffer.java:723) ~[?:1.8.0_362]
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:1083) ~[zookeeper-3.5.7.jar:3.5.7]
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1115) ~[zookeeper-3.5.7.jar:3.5.7] 

Here's the structure of my cluster
N0: ZK0, HMaster
N1: ZK1, Regionserver1
N1: ZK2, Regionserver2
N_100: HDFS

This error happen when I upgrade the HBase cluster, the zookeeper cluster also gets a restart. 
The error message happens rarely. Considering its ERROR level, I am not sure whether it will cause other issues. But the cluster still seems to be working correctly. I noticed that the send() code remains the same in the new version. I suspect it might also happen in the latest version. If it's benign, would it be better to be output as WARN level?

I have attached my full logs (persistent.tar.gz). The specific error occurred in hbase--zookeeper-8db357045302.log.