[IOTDB-918] [Distributed] Bind TsFiles with log index to accelerate snapshots

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Core/Cluster
• Labels: snapshot
• Assignee: Tian Jiang
• Reporter: Tian Jiang
• Created: 27/Sep/20 08:15
• Updated: 08/Jan/21 03:26
• Resolved: 08/Jan/21 03:26

Description

During snapshot catch-up, the leader will collect information of all TsFiles as RemoteTsFileResources, and send them to the followers. The follower will check if it has the file and decide whether to pull it from the leader.

To be smarter, the leader may be able to know that a TsFile of a certain version only contains data precedes to a certain log index, and the leader can avoid sending files that would be surely on the receiver side.

For example, when a TsFile `3.tsfile` is closed and the current committed log index is 100, the leader knows for sure that the file will not contain data corresponding to logs whose indexes are more than 100. When a follower whose committed log index exceeds 100 applies for a snapshot, the leader can skip `3.tsfile` since the follower must have it.

Issue Links

• is a child of: IOTDB-68 IoTDB supports distributed version