Uploaded image for project: 'Community Development'
  1. Community Development
  2. COMDEV-403

Apache IoTDB: Metadata (Schema) Storage Engine

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Apache IoTDB [1] is an Open Source IoT database designed to meet the rigorous data, storage, and analytics requirements of large-scale Internet of Things (IoT) and Industrial Internet of Things (IIoT) applications.

      Different with traditional relational databases, IoTDB uses tree-based structure in memory to manage the schema (a.k.a, metadata), and use a Write-Ahead-Log-like file structure to persist the schema.

      Now, each time series will take about 300 Bytes in memory. However, an IoTDB instance may manage more than 100 million time series, which may take more than 30GB memory.

      Therefore, we'd like to re-design the schema management module.
      1. File: Persist the tree on disk like a b-tree.
      2. WAL: implement the WAL of the metadata. So we can update the tree on disk in batch, rather than one operation by one.
      3. Cache: we may have no memory to load the whole tree. So a cache is needed and query from the tree on disk is needed.

      What knowledge you need to know:
      1. Java
      2. Basic design idea about Database [2]

      [1] https://iotdb.apache.org
      [2] http://pages.cs.wisc.edu/~dbbook/openAccess/firstEdition/slides/pdfslides/mod2l1.pdf

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            hxd Xiangdong Huang

            Dates

              Created:
              Updated:

              Slack

                Issue deployment