[RATIS-2129] Low replication performance because of lock contention on RaftLog

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: server
• Labels: Performance performance
• Assignee: Tsz-wo Sze
• Reporter: Duong
• Created: 22/Jul/24 22:10
• Updated: 8 hours ago
• Resolved: 30/Aug/24 17:40

Description

Today, the GrpcLogAppender thread makes a lot of calls that need RaftLog's readLock. In an active environment, RaftLog is always busy appending transactions from clients, thus writeLock is frequently busy. This makes the replication performance slow. 

See the dn_echo_leader_profile.html, or in the picture below, the purple is the time taken to acquire readLock from RaftLog.

1. 

A summary of LockContention in Ratis. 

 

Today, RaftLog consistency is protected by a global ReadWriteLock. (global means RaftLog has a single ReadWriteLock and the lock is acquired at the scope of the RaftLog instance, or a RaftGroup).

In a RaftGroup, the following actors race to obtain this global ReadWriteLock in the leader node:

• The writer, which is the GRPC Client Service, accepts transaction submissions from Raft clients and appends transactions (or log entries) to RaftLog. Each append operation needs to acquire the writeLock from RaftLog to put the transaction to RaftLog's memory queue. Although each of these append operations is quick, Ratis is designed to maximize transactions append and so the writeLock should be always busy.
• StateMachineUpdater. For each transaction, when it is acknowledged by enough followers, this single thread actor will read the log from RaftLog and call StateMachine to apply the transaction. This actor acquires readLock from RaftLog for each log entry read. 
• GrpcLogAppender: for each follower, there's a thread of GrpcLogAppender that constantly reads log entries from RaftLog and replicates them to the follower. This thread acquires readLock from RaftLog every time it reads a log entry.

All writer, StateMachineUpdater, and GrpcLogAppender are all designed in a way to maximize their throughput. For instance,  StateMachineUpdater invokes StateMachine's applyTransaction as asynchronous calls. The same is the way GrpcLogAppender replicates log entries to the follower. 

The global ReadWriteLock creates a tough contention between the RaftLog writers and readers. And that's what limit the ratis throughput down. The faster the writers and readers are, the more they block each other.

 

Issue Links

• causes: RATIS-2192 Lots of errors after applying RATIS-2129
• is caused by: RATIS-2099 Cache TermIndexImpl instead of using anonymous class
• is related to: RATIS-2151 TestRaftWithGrpc keeps failing with zero-copy
• links to: GitHub Pull Request #1127
• links to: GitHub Pull Request #1141