[RATIS-1597] Delay snapshot MD5 computing to InstallSnapshot stream process

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4.0, 3.0.0
• Component/s: performance
• Assignee: Song Ziyang
• Reporter: Song Ziyang
• Created: 24/Jun/22 04:02
• Updated: 16/Aug/22 16:34
• Resolved: 02/Jul/22 14:02

Description

Leader’s LogAppender while-loop checks latest snapshot info to decide wether to send a snapshot to a follower. The SnapshotInfo includes every snapshot file with its MD5 digest. Therefore, StateMachine is required to compute MD5 each time it takes a snapshot. 
 
However, for database workload, snapshot files may contain GBs of data, which makes MD5 computing a very consuming task. Since MD5 is only used when leader InstallSnapshot to a follower, it is better to compute MD5 along with InstallSnapshot stream process.
 
Currently, InstallSnapshot stream process will break snapshot file into fixed-size chunks and send them to followers one by one. Is it possible to calculate MD5 when reading each chunk? This implementation can avoid precomputing MD5 and minimize the IO cost.

Issue Links

• links to: GitHub Pull Request #661