[IOTDB-915] [Distributed] Merge StableEntryManager into CommittedLogManager

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: master branch
• Component/s: Core/Cluster
• Labels: cluster log-management persistency pull-request-available
• Assignee: Houliang Qi
• Reporter: Tian Jiang
• Created: 27/Sep/20 01:40
• Updated: 15/Jan/21 07:27
• Resolved: 15/Jan/21 07:26

Description

The current version of CommittedLogManager only holds a small number of logs, as it only uses the persistent storage (namely the StableEntryManager) as a measure of recovery, not to extend its storage ability. The result is that when one follower is down, its difference from the leader will soon exceed the capability of the CommittedLogManager, which further results in a snapshot catch-up, and it is time-consuming.

The idea is to merge StableEntryManager into CommittedLogManager, all committed logs should go to the persistent storage (unless persistency is disabled), and only the newest part of the logs will stay in memory. And CommittedLogManager should be able to read logs from the disk for catch-ups, provided they are not physically removed. Thus, the retention policy of memory logs and disk logs should be reconsidered.

Issue Links

• is a child of: IOTDB-68 IoTDB supports distributed version
• links to: GitHub Pull Request #1865