[KYLIN-1178] Build dictionary in Hadoop cluster

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Job Engine
• Assignee: Shao Feng Shi
• Reporter: Shao Feng Shi
• Created: 30/Nov/15 02:38
• Updated: 09/Aug/17 05:43
• Resolved: 13/Dec/16 08:41

Description

Kylin build dictionary in job engine node, usually this is okay. But if there is some high cardinality dimentions, the JVM heap couldn't fit in all distinct values, then job engine instance will crash with OOM error.

Need to enhance on this, move the dictionary building to another process or a hadoop node. Ideally only need modify "CreateDictionaryJob.java", move the dictionary building to a mapper task.

Issue Links

• Is contained by: KYLIN-1851 Improve build dictionary, consider that input is already sorted
• is related to: KYLIN-943 Approximate TopN supported by Cube