[KYLIN-4369] Build Global Dict by MR/Hive,Set Config

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Sprint: Sprint 50
• Assignee: wangxiaojing
• Reporter: wangxiaojing
• Created: 03/Feb/20 07:22
• Updated: 30/May/20 11:17
• Resolved: 30/May/20 11:17

Description

Enable Mr / hive global dictionary

1.Set the measures to count distinct and select the precise type column.

2.Set cube —— Configuration Overwrites:
Key：kylin.dictionary.mr-hive.columns
Value: the alias of the global dictionary column is required. Multiple columns are separated by commas,Eg:KYLIN_SALES_SELLER_ID,KYLIN_SALES_BUYER_ID

Configure the number of reduce for each global dict column

Not necessary.By default, there are two reduce for each column. The more the number, the higher the concurrency.

1.Set cube —— Configuration Overwrites
Key：kylin.dictionary.mr-hive.columns.reduce.num
Value: the alias of reduce dictionary column needs to be set: the number of reduce columns, separated by commas, Eg: KYLIN_SALES_SELLER_ID:3

Use global domain dictionary

Reuse Mr/Hive global dictionary columns of other cubes to avoid building dictionaries repeatedly

1.cube——Configuration Overwritese
Key: kylin.dictionary.mr-hive.ref.columns
Value: globalDictColumnName:referencedCubeName_referencedColumnName, multiple columns separated by commas,Eg: KYLIN_SALES_BUYER_ID global MR/Hive Dict reuse cube kylin_sales_cube_spark_notopn's global MR/Hive column KYLIN_SALES_BUYER_ID, we can config: KYLIN_SALES_BUYER_ID:kylin_sales_cube_spark_notopn.KYLIN_SALES_BUYER_ID