[IOTDB-860] [Distributed]Emend the async log applier

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Core/Cluster
• Labels: pull-request-available
• Assignee: Houliang Qi
• Reporter: Houliang Qi
• Created: 01/Sep/20 07:07
• Updated: 30/Oct/20 02:35
• Resolved: 08/Sep/20 02:35

Description

Now we support the async log applier to make the inset operations parallel in different storage groups, at the same time, we need to rethink the following things when do snapshot:

1. When starting to take a snapshot, record the current commitIndex.
2. Block until all logs whose indices <= the recorded commitIndex are applied. (Use RaftLogManager to do so instead of LogApplier)
3. Prevent the log cleaner thread to clean logs that are not applied.
4. Change StorageEngine.getInstance().syncCloseAllProcessor(); in takeSnapshot() to send a flush plan within the group. (So the file sequence will not be broken by snapshots)
5. When committed logs are recovered during start-up, re-apply all of them. (Notice that operation sequences in IoTDB are idempotent)

 

Issue Links

• is a child of: IOTDB-68 IoTDB supports distributed version
• links to: GitHub Pull Request #1635