[ZOOKEEPER-3179] Add snapshot compression to reduce the disk IO

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Labels: pull-request-available
• Assignee: Yisong Yue
• Reporter: Fangmin Lv
• Created: 19/Oct/18 21:45
• Updated: 09/Mar/21 18:20
• Resolved: 10/Apr/19 11:19

Description

When the snapshot becomes larger, the periodically snapshot after certain number of txns will be more expensive. Which will in turn affect the maximum throughput we can support within SLA, because of the disk contention between snapshot and txn when they're on the same drive.
 
With compression like zstd/snappy/gzip, the actual snapshot size could be much smaller, the compress ratio depends on the actual data. It might make the recovery time (loading from disk) faster in some cases, but will take longer sometimes because of the extra time used to compress/decompress.
 
Based on the production traffic, the performance various with different compress method as well, that's why we provided different implementations, we can select different compress method for different use cases.
 

Issue Links

• duplicates: ZOOKEEPER-1651 Add support for compressed snapshot
• is related to: ZOOKEEPER-4231 Add document for snapshot compression config
• links to: GitHub Pull Request #690
• links to: GitHub Pull Request #690