Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-562

[Distributed]Apache IoTDB raft log persistence in the distributed version

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core/Cluster

      Description

      IoTDB is a highly efficient time series database, which supports high-speed query process, including aggregation query.

      Currently, IoTDB has supported shared-nothing cluster which using raft mechanism and raft logs to communicate among all nodes. So raft logs are very important in communication, consistency keeping, and fail-over.

      However, the current logs are just stored in memory which means raft logs will lost when the nodes down and then recover. Secondly, the raft logs may overlap with current WAL which means we may do some unnecessary log writing works.

      So there are two improvements about raft logs need to be done:

      1. Store the raft logs in a durable material such as a disk. You need to design a serializable form of logs and then put them to disk.

      2. Find a way of using raft logs in the IoTDB recovery process. That's means we just write raft logs rather than both raft logs and WAL. This will avoid some unnecessary log writing works and improve insertion performance.

      This proposal is mainly for improving raft logs in clustered IOTDB. Besides, if we can let the summary info be more useful, it could be better.

      Notice that the premise is that the raft logs writing process should not be slow down too much. That means the serializable form should be high efficiency enough.

      You should know:
      • IoTDB cluster structure
      • IoTDB WAL
      • IoTDB insertion process
      • Raft
      • Java

      difficulty: Major
      mentors:
      jt2594838@163.com

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                SilverNarcissus Kaifeng Xue
                Reporter:
                jt2594838 Tian Jiang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: