Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16520 Cache hive metadata in metastore
  3. HIVE-19126

CachedStore: Use memory estimation to limit cache size during prewarm

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0, 3.1.0
    • Component/s: Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We can rely on https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java to estimate memory of SharedCache. This jira addresses the size estimation during prewarm, so that we can stop when we hit the memory limit. In a follow-up jira, we will work on estimation/eviction after prewarm is complete, so that we can keep the frequently used tables and their partitions in cache.

        Attachments

        1. HIVE-19126.6.patch
          74 kB
          Vaibhav Gumashta
        2. HIVE-19126.5.patch
          71 kB
          Vaibhav Gumashta
        3. HIVE-19126.5.patch
          74 kB
          Vaibhav Gumashta
        4. HIVE-19126.4.patch
          71 kB
          Vaibhav Gumashta
        5. HIVE-19126.3.patch
          71 kB
          Vaibhav Gumashta
        6. HIVE-19126.2.patch
          73 kB
          Vaibhav Gumashta
        7. HIVE-19126.1.patch
          68 kB
          Vaibhav Gumashta

          Issue Links

            Activity

              People

              • Assignee:
                vgumashta Vaibhav Gumashta
                Reporter:
                vgumashta Vaibhav Gumashta
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: