Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1844

Hold huge dictionary in 2nd storage like disk/hbase

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • v1.2, v1.5.2
    • None
    • Query Engine
    • None

    Description

      A whole dimension is kept in memory.

      We should have a way to keep only certain number / size of total rows to be kept in memory. A LRU cache for rows in the dimension will help keep memory in check.

      Why not store all the dimensions data in hbase in a different table with a prefix of dimensionid, and all calls to the dimensions (get based on dim key), is mapped to hbase.

      This does mean it will cost more time on a miss.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            liyang.gmt8@gmail.com liyang
            llabhilash Abhilash L L
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment