Uploaded image for project: 'Community Development'
  1. Community Development
  2. COMDEV-403

Apache IoTDB: Metadata (Schema) Storage Engine

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments



      Apache IoTDB [1] is an Open Source IoT database designed to meet the rigorous data, storage, and analytics requirements of large-scale Internet of Things (IoT) and Industrial Internet of Things (IIoT) applications.

      Different with traditional relational databases, IoTDB uses tree-based structure in memory to manage the schema (a.k.a, metadata), and use a Write-Ahead-Log-like file structure to persist the schema.

      Now, each time series will take about 300 Bytes in memory. However, an IoTDB instance may manage more than 100 million time series, which may take more than 30GB memory.

      Therefore, we'd like to re-design the schema management module.
      1. File: Persist the tree on disk like a b-tree.
      2. WAL: implement the WAL of the metadata. So we can update the tree on disk in batch, rather than one operation by one.
      3. Cache: we may have no memory to load the whole tree. So a cache is needed and query from the tree on disk is needed.

      What knowledge you need to know:
      1. Java
      2. Basic design idea about Database [2]

      [1] https://iotdb.apache.org
      [2] http://pages.cs.wisc.edu/~dbbook/openAccess/firstEdition/slides/pdfslides/mod2l1.pdf



          This comment will be Viewable by All Users Viewable by All Users


            Unassigned Unassigned
            hxd Xiangdong Huang




                Issue deployment