Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-918

[Distributed] Bind TsFiles with log index to accelerate snapshots

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core/Cluster
    • Labels:

      Description

      During snapshot catch-up, the leader will collect information of all TsFiles as RemoteTsFileResources, and send them to the followers. The follower will check if it has the file and decide whether to pull it from the leader.

      To be smarter, the leader may be able to know that a TsFile of a certain version only contains data precedes to a certain log index, and the leader can avoid sending files that would be surely on the receiver side.

      For example, when a TsFile `3.tsfile` is closed and the current committed log index is 100, the leader knows for sure that the file will not contain data corresponding to logs whose indexes are more than 100. When a follower whose committed log index exceeds 100 applies for a snapshot, the leader can skip `3.tsfile` since the follower must have it.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jt2594838 Tian Jiang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: