[IOTDB-998] [Distributed] Split the snapshot when sending to follower

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Core/Cluster
• Assignee: Unassigned
• Reporter: Houliang Qi
• Created: 11/Nov/20 06:16
• Updated: 13/Nov/20 02:00

Description

Now, when the follower is much behind the leader, the leader will send a snapshot, but when the snapshot size is greater than thrift's maxFrameSize, the sending will fail, so that the follower will never catch up with the leader.

I think we need to split the snapshot request and then send it to the follower. But this also brings a problem when we use an asynchronous server. How to ensure the order after the split request is sent to the follower. Two options are possible as follows:
1. Mark the segment number in the header of the request sent by the leader, and splice the request on the follower side.
2. Like the follower downloading TsFile process, the snapshot is downloaded in segments on the follower side, but the snapshot data needs to be persisted on the leader side.

I would like the first one, any ideas about this?

Issue Links

• is a child of: IOTDB-68 IoTDB supports distributed version
• is part of: IOTDB-1005 [Distributed] Make the frame size of thrift scalable