[IOTDB-562] [Distributed]Apache IoTDB raft log persistence in the distributed version

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Core/Cluster
• Labels: IoTDB gsoc2020 mentor
• Assignee: Kaifeng Xue
• Reporter: Tian Jiang
• Created: 19/Mar/20 04:48
• Updated: 30/Oct/20 02:37
• Resolved: 20/May/20 04:00

Description

IoTDB is a highly efficient time series database, which supports high-speed query process, including aggregation query.

Currently, IoTDB has supported shared-nothing cluster which using raft mechanism and raft logs to communicate among all nodes. So raft logs are very important in communication, consistency keeping, and fail-over.

However, the current logs are just stored in memory which means raft logs will lost when the nodes down and then recover. Secondly, the raft logs may overlap with current WAL which means we may do some unnecessary log writing works.

So there are two improvements about raft logs need to be done:

1. Store the raft logs in a durable material such as a disk. You need to design a serializable form of logs and then put them to disk.

2. Find a way of using raft logs in the IoTDB recovery process. That's means we just write raft logs rather than both raft logs and WAL. This will avoid some unnecessary log writing works and improve insertion performance.

This proposal is mainly for improving raft logs in clustered IOTDB. Besides, if we can let the summary info be more useful, it could be better.

Notice that the premise is that the raft logs writing process should not be slow down too much. That means the serializable form should be high efficiency enough.

You should know:
• IoTDB cluster structure
• IoTDB WAL
• IoTDB insertion process
• Raft
• Java

difficulty: Major
mentors:
[e-mail]

Issue Links

• contains: IOTDB-351 [Distributed] Serialize the raft logs
• is a child of: IOTDB-68 IoTDB supports distributed version