[KYLIN-3729] CLUSTER BY CAST(field AS STRING) will accelerate base cuboid build with UHC global dict

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: v2.5.2
• Fix Version/s: v2.6.0
• Component/s: Job Engine
• Assignee: Fangyuan Deng
• Reporter: Fangyuan Deng
• Created: 19/Dec/18 03:53
• Updated: 27/Jan/19 14:57
• Resolved: 26/Dec/18 06:13

Description

As we know global dict is a sliced  appendTrieTree using cache-loader , so if we convert values to ids using global dict, ordered values will help.

And now we can set kylin.source.hive.flat-table-cluster-by-dict-column = uhc column, to make source data CLUSTER BY uhc-column, this get better.

But the appendTrieTree is order by string, so we can  CLUSTER BY CAST(uhc-column AS STRING), to optimize most.

We can see the hdfs bytes read (most is global dict) reduce to 30%