Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16520 Cache hive metadata in metastore
  3. HIVE-17495

CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Metastore
    • None
    • Reviewed

    Description

      Only when CachedStore is enabled, we would like to make the following optimizations:
      1. During CachedStore prewarm, use one sql call to retrieve column stats objects for a db and store it in the cache.
      2. Cache some aggregate stats (e.g. aggregate stats for all partitions, which seems to be commonly used) for query compilation speedup.
      3. There was a bug in MetaStoreUtils.aggrPartitionStats, which would use an iterator.next w/o checking with iterator.hasNext. This patch refactors some code to fix that.

      Attachments

        1. HIVE-17495.1.patch
          99 kB
          Vaibhav Gumashta
        2. HIVE-17495.10.patch
          108 kB
          Vaibhav Gumashta
        3. HIVE-17495.10.patch
          108 kB
          Vaibhav Gumashta
        4. HIVE-17495.2.patch
          137 kB
          Vaibhav Gumashta
        5. HIVE-17495.3.patch
          167 kB
          Vaibhav Gumashta
        6. HIVE-17495.4.patch
          122 kB
          Vaibhav Gumashta
        7. HIVE-17495.5.patch
          121 kB
          Vaibhav Gumashta
        8. HIVE-17495.6.patch
          121 kB
          Vaibhav Gumashta
        9. HIVE-17495.7.patch
          121 kB
          Vaibhav Gumashta
        10. HIVE-17495.8.patch
          121 kB
          Vaibhav Gumashta
        11. HIVE-17495.9.patch
          121 kB
          Vaibhav Gumashta

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vgumashta Vaibhav Gumashta Assign to me
            vgumashta Vaibhav Gumashta
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment