[ZOOKEEPER-2977] Concurrency for addAuth corrupts quorum packets

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Abandoned
• Affects Version/s: 3.4.9
• Component/s: quorum
• Labels: pull-request-available

Environment

Affects all version in 3.4.x

• Tags: Quorum, shutdown
• Assignee: Sumit Agrawal
• Reporter: Sumit Agrawal
• Created: 12/Feb/18 03:40
• Updated: 17/May/23 12:47
• Resolved: 17/May/23 12:47

Description

When client performs multiple times addAuth with different credential at follower concurrently, the communication between follower gets corrupt. This causes shutdown of Follower due to the failure.

Analysis:

In org.apache.zookeeper.server.quorum.QuorumPacket.serialize method,

• call a_.startVector(authinfo,"authinfo"); which write the length of authinfo to packet (suppose it writes length 1)
• get length of authinfo to write all details in loop (here gets length as 2)

<-- Here in concurrency scenario, buffer gets corrupt having extra bytes in channel for additional authinfo.

 

So When Leader reads next quorum packet, it reads previous extra bytes (incorrect) and possibly identify greater size of message (as corrupt byte pattern) causes exception...

Coordination > Unexpected exception causing shutdown while sock still open (LearnerHandler.java:633)
java.io.IOException: Unreasonable length = 1885430131

 

 

ServerCnxn.getAuthInfo returns Unmodifiable list, but while addAuthInfo, there is no check. So this causes concurrency issue.

 

 

 

 

Issue Links

• is fixed by: ZOOKEEPER-1525 Plumb ZooKeeperServer object into auth plugins
• links to: GitHub Pull Request #474
• links to: GitHub Pull Request #474