Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16520 Cache hive metadata in metastore
  3. HIVE-18264

CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None
    • Reviewed


      Currently we have a separate cache for partitions and partition col stats which results in some calls iterating through each of these for retrieving/updating. For example, to modify a partition col stat, currently we need to lock table, partition and partition col stats caches which are all separate hashmaps. We can get better performance by organizing hierarchically. For example, we can have a partition, partition col stats and table col stats cache per table to improve on the previous mechanisms. This will also result in better concurrency, since now instead of locking the whole cache, we can selectively lock the table cache and modify multiple tables in parallel.

      In addition, currently, the prewarm mechanism populates all the caches initially (it skips tables that do not pass whitelist/blacklist filter) and it is a blocking call. This patch also makes prewarm non-blocking so that the calls for tables that are already cached can be served from the memory and the ones that are not can be served from the rdbms.


        1. HIVE-18264.1.patch
          123 kB
          Vaibhav Gumashta
        2. HIVE-18264.2.patch
          163 kB
          Vaibhav Gumashta
        3. HIVE-18264.3.patch
          197 kB
          Vaibhav Gumashta
        4. HIVE-18264.4.patch
          205 kB
          Vaibhav Gumashta
        5. HIVE-18264.5.patch
          213 kB
          Vaibhav Gumashta
        6. HIVE-18264.6.patch
          224 kB
          Vaibhav Gumashta
        7. HIVE-18264.7.patch
          224 kB
          Vaibhav Gumashta
        8. HIVE-18264.8.patch
          224 kB
          Vaibhav Gumashta
        9. HIVE-18264.8.patch
          224 kB
          Vaibhav Gumashta


          This comment will be Viewable by All Users Viewable by All Users


            vgumashta Vaibhav Gumashta Assign to me
            vgumashta Vaibhav Gumashta
            0 Vote for this issue
            5 Start watching this issue




                Issue deployment