[KYLIN-4366] Build Global Dict by MR/Hive, Merge to dict table Step implementation

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Sprint: Sprint 50
• Assignee: wangxiaojing
• Reporter: wangxiaojing
• Created: 02/Feb/20 14:58
• Updated: 30/May/20 10:30
• Resolved: 30/May/20 10:30

Description

Merge the global dict values of each column calculated by previous parallel total build step into the global dictionary table (default globalDictDatabase.cubeName_global_dict) of this cube.

HQL example below:

//Add partition if not exists
ALTER TABLE kylin_intermediate_kylin_sales_cube_mr_6222c210_ce2d_e8ce_dd0f_f12c38fa9115_global_dict add  IF NOT EXISTS partition (dict_column='KYLIN_SALES_SELLER_ID'); 
ALTER TABLE kylin_intermediate_kylin_sales_cube_mr_6222c210_ce2d_e8ce_dd0f_f12c38fa9115_global_dict add  IF NOT EXISTS partition (dict_column='KYLIN_SALES_BUYER_ID');

//merge data
INSERT OVERWRITE TABLE db.kylin_sales_cube_mr_global_dict PARTITION (dict_column = 'KYLIN_SALES_SELLER_ID') 
SELECT dict_key, dict_val FROM db.kylin_sales_cube_mr_global_dict WHERE dict_column = 'KYLIN_SALES_SELLER_ID' 
UNION 
SELECT dict_key, dict_val FROM kylin_intermediate_kylin_sales_cube_mr_6222c210_ce2d_e8ce_dd0f_f12c38fa9115_global_dict WHERE dict_column = 'KYLIN_SALES_SELLER_ID' ;
INSERT OVERWRITE TABLE db.kylin_sales_cube_mr_global_dict PARTITION (dict_column = 'KYLIN_SALES_BUYER_ID') 
SELECT dict_key, dict_val FROM db.kylin_sales_cube_mr_global_dict WHERE dict_column = 'KYLIN_SALES_BUYER_ID' 
UNION 
SELECT dict_key, dict_val FROM kylin_intermediate_kylin_sales_cube_mr_6222c210_ce2d_e8ce_dd0f_f12c38fa9115_global_dict WHERE dict_column = 'KYLIN_SALES_BUYER_ID' ;

Issue Links

• links to: GitHub Pull Request #1093