Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4141

Build Global Dictionary in no time

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v3.0.0-beta
    • Fix Version/s: v3.0.0-beta
    • Component/s: Real-time Streaming
    • Labels:
      None

      Description

      Backgroud

      Currently, realtime OLAP do not support COUNT_DISTINCT(bitmap) for non interger type.
      Because of the lack the ability of encoding string at once, so I want to use RocksDB & HBase as implementation of streaming distributed dictionary. 

      Design

      1. each receiver will own a local dict cache
      2. all receiver will share a remote dict storage
      3. we choose to use RocksDB as local dict cache
      4. we choose to use HBase as remote dict storage
         
      5. for each cube, we will create a local dict and a hbase table
      6. we will create column family both in RocksDB and HBase for each column which occur in COUNT_DISTINCT

      Design Diagram

       

        Attachments

        1. image-2019-09-20-19-04-47-937.png
          46 kB
          Xiaoxiang Yu
        2. image-2019-09-20-19-04-55-935.png
          99 kB
          Xiaoxiang Yu
        3. image-2019-09-20-20-06-15-960.png
          144 kB
          Xiaoxiang Yu
        4. image-2019-09-22-14-54-07-772.png
          69 kB
          Xiaoxiang Yu
        5. image-2019-09-22-16-10-41-831.png
          11 kB
          Xiaoxiang Yu
        6. image-2019-09-22-16-14-15-963.png
          184 kB
          Xiaoxiang Yu
        7. image-2019-09-22-16-28-54-593.png
          161 kB
          Xiaoxiang Yu
        8. image-2019-09-22-20-40-06-476.png
          168 kB
          Xiaoxiang Yu
        9. image-2019-09-22-20-41-13-146.png
          13 kB
          Xiaoxiang Yu
        10. mock_message.py
          7 kB
          Xiaoxiang Yu

          Issue Links

            Activity

              People

              • Assignee:
                hit_lacus Xiaoxiang Yu
                Reporter:
                hit_lacus Xiaoxiang Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: