[KYLIN-1844] Hold huge dictionary in 2nd storage like disk/hbase

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: v1.2, v1.5.2
• Component/s: Query Engine
• Assignee: liyang
• Reporter: Abhilash L L
• Created: 01/Jul/16 06:50
• Updated: 27/May/20 13:42
• Resolved: 27/May/20 13:42

Description

A whole dimension is kept in memory.

We should have a way to keep only certain number / size of total rows to be kept in memory. A LRU cache for rows in the dimension will help keep memory in check.

Why not store all the dimensions data in hbase in a different table with a prefix of dimensionid, and all calls to the dimensions (get based on dim key), is mapped to hbase.

This does mean it will cost more time on a miss.

Issue Links

• relates to: KYLIN-1428 Scalable dictionary to support cardinality of billions