[KYLIN-3491] Improve the cube building process when using global dictionary

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v2.5.0
• Component/s: Job Engine
• Assignee: Zhong Yanghong
• Reporter: Zhong Yanghong
• Created: 10/Aug/18 09:04
• Updated: 19/Mar/21 09:15
• Resolved: 03/Sep/18 10:52

Description

By current cubing process, if the global dictionary is very large, since the raw data records are unsorted, it's hard to encode raw values into ids for the input of bitmap due to frequent swap of the dictionary slices. We need a refined process. The idea is as follows:

1. for each source data block, there will be a mapper generating the distinct values & sort them
2. encode the sorted distinct values and generate a shrunken dict for each source data block.
3. when building base cuboid, use the shrunken dict for each source data block for encoding.

Issue Links

• is a parent of: KYLIN-4940 Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
• relates to: KYLIN-3424 Missing invoke addCubingGarbageCollectionSteps in the cleanup step for HBaseMROutput2Transition