[ZOOKEEPER-2933] Ability to monitor the jute.maxBuffer usage in real-time

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0, 3.5.5
• Component/s: jute, server
• Labels: buffer buffer-length
• Assignee: Andor Molnar
• Reporter: Andor Molnar
• Created: 06/Nov/17 16:22
• Updated: 20/May/19 17:50
• Resolved: 02/Oct/18 09:38

Description

This is related to jute.maxbuffer problems on the server side when Leader generates a proposal that doesn't fit into Follower's Jute buffer causing the quorum to be broken.

Proposed solution is to add the following new JMX Beans:

1. Add getJuteMaxBuffer to ZookeeperServerBean which monitors the current jute.maxbuffer setting,
2. Add get last/min/max ProposalSize to LeaderBean which monitors the size of the latest/min/max proposal.

The rationale behind this new feature is to add capability to JMX monitoring API to determine what is the current/min/max usage of the Jute buffer. This will let third party monitoring tools to get samples of buffer usage and create some statistics or generate alerts if it breaches a particular value.

This will not solve the problems related to jute.maxbuffer setting on its own, but it's intended to be the first step towards better handling or preventing production issues to happen.

Subtasks have been created to separately implement client and server side buffer size monitoring.

Issue Links

• links to: GitHub Pull Request #415