[KYLIN-5019] Avoid building global dictionary from all data of fact table each time

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v4.0.0-beta
• Fix Version/s: v4.0.0
• Assignee: Shengjun Zheng
• Reporter: Shengjun Zheng
• Created: 28/Jun/21 10:01
• Updated: 24/Aug/21 07:17
• Resolved: 24/Aug/21 07:17

Description

Global dictionaries will be resized to avoid single dictionary bucket exceeding its threashold in each cubing job. To judge whether a dictionary should be resized, Kylin will read all data from fact table and do count distinct operation. Then rebuild dictionary from all history data.

It’s better to use a lightweight approach, since dictionary don't need to be resized for incremental building in most cases.