[ZOOKEEPER-2684] Fix a crashing bug in the mixed workloads commit processor

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.6.0
• Fix Version/s: 3.6.0
• Component/s: server
• Labels: pull-request-available

Environment

with pretty heavy load on a real cluster

• Assignee: Kfir Lev-Ari
• Reporter: Ryan Zhang
• Created: 07/Feb/17 05:23
• Updated: 30/Jan/19 18:46
• Resolved: 03/Nov/17 04:20

Description

We deployed our build with ZOOKEEPER-2024 and it quickly started to crash with the following error

atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:24:42,305 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x119fa expected 0x11fc5 for client session id 1009079ba470055
atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:32:04,746 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x698 expected 0x928 for client session id 4002eeb3fd0009d
atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:34:46,648 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x8904 expected 0x8f34 for client session id 51b8905c90251
atla-buh-05-sr1.prod.twttr.net: 2017-01-18 22:43:46,834 - ERROR [CommitProcessor:2] -org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:268) – Got cxid 0x3a8d expected 0x3ebc for client session id 2051af11af900cc

clearly something is not right in the new commit processor per session queue implementation.

Issue Links

• Blocked: ZOOKEEPER-2024 Major throughput improvement with mixed workloads
• links to: GitHub Pull Request #167
• links to: GitHub Pull Request #167
• links to: GitHub Pull Request #411