Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16520 Cache hive metadata in metastore
  3. HIVE-17495

CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Metastore
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Only when CachedStore is enabled, we would like to make the following optimizations:
      1. During CachedStore prewarm, use one sql call to retrieve column stats objects for a db and store it in the cache.
      2. Cache some aggregate stats (e.g. aggregate stats for all partitions, which seems to be commonly used) for query compilation speedup.
      3. There was a bug in MetaStoreUtils.aggrPartitionStats, which would use an iterator.next w/o checking with iterator.hasNext. This patch refactors some code to fix that.

        Attachments

        1. HIVE-17495.10.patch
          108 kB
          Vaibhav Gumashta
        2. HIVE-17495.10.patch
          108 kB
          Vaibhav Gumashta
        3. HIVE-17495.9.patch
          121 kB
          Vaibhav Gumashta
        4. HIVE-17495.8.patch
          121 kB
          Vaibhav Gumashta
        5. HIVE-17495.7.patch
          121 kB
          Vaibhav Gumashta
        6. HIVE-17495.6.patch
          121 kB
          Vaibhav Gumashta
        7. HIVE-17495.5.patch
          121 kB
          Vaibhav Gumashta
        8. HIVE-17495.4.patch
          122 kB
          Vaibhav Gumashta
        9. HIVE-17495.3.patch
          167 kB
          Vaibhav Gumashta
        10. HIVE-17495.2.patch
          137 kB
          Vaibhav Gumashta
        11. HIVE-17495.1.patch
          99 kB
          Vaibhav Gumashta

          Issue Links

            Activity

              People

              • Assignee:
                vgumashta Vaibhav Gumashta
                Reporter:
                vgumashta Vaibhav Gumashta
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: