Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3905

Enable shrunken dictionary default

    XMLWordPrintableJSON

    Details

      Description

      In dev mail list's discussion, I suggest to enable shrunken dictionary by default, and received some dev's aggrement. 

      When using bitmap measure on a large cardinality column(require global dictionaty), build base cuboid step need frequent cache swap so it cannot finished within a reasonable period.
      When shrunken dictionary enabled, a new step will be added to build separated dictionary for each `InputSplit`, Mapper of *BuildBaseCuboid* step only has to fetch a smaller dictionary for itself, instead of a larger global dictionary. It will reduce cache swap and make *BuildBaseCuboid* step run as quicker as possible.

       

      http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62EFCB72-B235-4FC3-9ADD-0FC510D97768@kyligence.io%3e

        Attachments

        1. image-2019-03-25-11-29-19-383.png
          427 kB
          Xiaoxiang Yu
        2. image-2019-03-25-11-28-14-256.png
          243 kB
          Xiaoxiang Yu
        3. image-2019-03-25-11-27-46-175.png
          134 kB
          Xiaoxiang Yu
        4. image-2019-03-25-11-27-26-149.png
          138 kB
          Xiaoxiang Yu
        5. image-2019-03-25-11-26-59-198.png
          135 kB
          Xiaoxiang Yu

          Issue Links

            Activity

              People

              • Assignee:
                hit_lacus Xiaoxiang Yu
                Reporter:
                hit_lacus Xiaoxiang Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: