Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-2210

Streaming ingestion interfaces to reduce headers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      When ingesting with an extremely small batch size, the tablet header may become a significant overhead. For example, if the batch size is 1 and the data type is double, in a tablet, each timeseries only has 16 bytes of data. The header (measurement name and data type) of each timeseries may also be around 10 bytes, so more than 1/3 of the bandwidth is wasted on transmitting headers. The small-batch situation is rather common when the sampling frequency is low or clients do not have enough capacity to batch data.

      To avoid wasting bandwidth on headers when headers are fixed, we propose a streaming ingestion interface by splitting tablets into headers and contents. The interface is composed of two functions:
      `long createTabletStream(String deviceId, List<IMeasurementSchema> schemas, int maxRowNum, boolean isAligned)`
      `void writeTabletStream(long tabletId, long[] timestamps, Object[] values, BitMap[] bitMaps)`

      Firstly `createTabletStream` is invoked by a client, then the server creates an empty tablet with the provided headers, assigns a unique id to it, and returns the id to the client. Next, with the tabletId and data to write, the client invokes `writeTabletStream`. The server finds the associated tablet with the tabletId, fills the tablet with data from the client, and inserts the tablet.

      Some exceptions that should be handled: the tabletId cannot be found by the server; the tablet content cannot fit the tablet; unused tablets should be cleaned...

      Attachments

        Activity

          People

            Unassigned Unassigned
            jt2594838 Tian Jiang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: