[GSOC-208] Apache IoTDB: use netty as the memory buffer pool to reduce GC problem and take full use of memory

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: IoTDB gsoc2021 mentor
• Assignee: Unassigned
• Reporter: Xiangdong Huang
• Created: 26/Mar/21 07:10
• Updated: 21/Mar/23 08:11

Description

Apache IoTDB [1] is an Open Source IoT database designed to meet the rigorous data, storage, and analytics requirements of large-scale Internet of Things (IoT) and Industrial Internet of Things (IIoT) applications.

 

Memory control is very very important for DBMS.

 

Currently, we are using a customized memory buffer pool, which contains a pool of int[], a pool of long[], a pool of boolean[] and as well as float[] and double[].

 

However, there are two things left:

1. it is complex to implement a buffer pool for String[] or byte[][], as the size of String and byte[] are variable.
2. We are using HeapByteBuffer, while in many cases, directByteBuffer is more efficient.

As Netty has provided a high efficient buffer pool, we'd like to try to migrate the current buffer pool to the Netty implementation.

 

Things you should know:

1. Know Java well
2. Know Netty well
3. read codes of IoTDB (mainly in StorageEngine, StorageGroupProcessor and related classes)