[KYLIN-4141] Build Global Dictionary in no time

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v3.0.0-beta
• Fix Version/s: v3.0.0-beta
• Component/s: Real-time Streaming
• Assignee: Xiaoxiang Yu
• Reporter: Xiaoxiang Yu
• Created: 19/Aug/19 13:54
• Updated: 05/Nov/19 06:22
• Resolved: 05/Nov/19 06:22

Description

Backgroud

Currently, realtime OLAP do not support COUNT_DISTINCT(bitmap) for non interger type.
Because of the lack the ability of encoding string at once, so I want to use RocksDB & HBase as implementation of streaming distributed dictionary. 

Design

1. each receiver will own a local dict cache
2. all receiver will share a remote dict storage
3. we choose to use RocksDB as local dict cache
4. we choose to use HBase as remote dict storage
    
5. for each cube, we will create a local dict and a hbase table
6. we will create column family both in RocksDB and HBase for each column which occur in COUNT_DISTINCT

Design Diagram

 

Issue Links

• links to: Code change