[KYLIN-3905] Enable shrunken dictionary default

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: v2.6.2, v3.0.0-alpha
• Component/s: Measure - Count Distinct
• Assignee: Xiaoxiang Yu
• Reporter: Xiaoxiang Yu
• Created: 25/Mar/19 03:22
• Updated: 03/Jul/20 03:00
• Resolved: 27/Mar/19 03:30

Description

In dev mail list's discussion, I suggest to enable shrunken dictionary by default, and received some dev's aggrement. 

When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period.
When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of *BuildBaseCuboid* step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make *BuildBaseCuboid* step run as quicker as possible.

 

http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/[e-mail]%3e

Issue Links

• is related to: KYLIN-4619 Make shrunken dict able to coexist with mr-hive global dict
• links to: GitHub Pull Request #549
• links to: GitHub Pull Request #765