[IGNITE-20818] High periodic latency spikes on long put-get benchmark

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Component/s: persistence
• Labels: ignite-3 ignite3_performance
• Assignee: Ivan Bessonov
• Reporter: Ivan Artiukhov
• Reviewer: Roman Puchkovskiy
• Created: 09/Nov/23 08:37
• Updated: 04/Sep/24 16:08
• Resolved: 21/Nov/23 07:15

Description

AI3, rev. c15f14f4ddca7d6e9ab3e02d5a661e68d6e9ee21

Benchmark parameters:

• 1 server node
• aipersist storage engine, 25 partitions (default), raft.fsync=false

Benchmark code: https://github.com/gridgain/YCSB/blob/ycsb-2023.7/ignite3/src/main/java/site/ycsb/db/ignite3/IgniteClient.java 

Benchmark parameters:

• 100% put of 25 million unique entries, 1 thread
• then 100% get of 25 million entries, 1 thread

"put" throughput:

"get" throughput:

On the graphs we see the performance drop which happen every 5 minutes which begins after ~50 minutes since the start of the test. Then after ~4 hours and 20 minutes the periodic drops stopped to appear. 

Timings:

12:11 -- puts start
13:01 -- first spike
15:46 -- puts end
15:47 -- gets start
16:31 -- spikes ended
18:27 -- gets end

 

Issue Links

• links to: GitHub Pull Request #2838