[KYLIN-2192] More Robust Global Dictionary

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v1.5.4.1
• Fix Version/s: v2.0.0
• Component/s: Job Engine
• Assignee: Yerui Sun
• Reporter: Yerui Sun
• Created: 15/Nov/16 12:36
• Updated: 05/Feb/18 09:25
• Resolved: 30/Mar/17 04:00

Description

Global dictionary have been released over 2 months, I've received some feedbacks and bug reports. Here's the patch to make global dictionary more robust, including some functional improvements.

• Break through 255 bytes limitation for value, but still recommend value length less than 8K, avoiding stack overflow error;
• Fix 'Value not exists' or stack overflow error when dict size is larger than 1GB, the root cause is similar with KYLIN-1834; A check tool also provided for check corrupted or not of existing dict data;
• Support parallel dictionary building in one job server, used for parallel segments building;

Issue Links

• is duplicated by: KYLIN-2352 java.lang.IllegalStateException: GlobalDict /dict/tableName/column should have 0 or 1 append dict but 2! - in Step 4 - Build Dimension Dictionary