Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16520

Cache hive metadata in metastore

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Metastore
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      To use CachedStore, please set hive.metastore.rawstore.impl to "org.apache.hadoop.hive.metastore.cache.CachedStore" in hive-site.xml.

      Description

      During Hive 2 benchmark, we find Hive metastore operation take a lot of time and thus slow down Hive compilation. In some extreme case, it takes much longer than the actual query run time. Especially, we find the latency of cloud db is very high and 90% of total query runtime is waiting for metastore SQL database operations. Based on this observation, the metastore operation performance will be greatly enhanced if we have a memory structure which cache the database query result.

        Attachments

        1. HIVE-16520-proto.patch
          83 kB
          Daniel Dai
        2. HIVE-16520-proto-2.patch
          84 kB
          Daniel Dai
        3. HIVE-16520-1.patch
          105 kB
          Daniel Dai
        4. HIVE-16520.2.patch
          107 kB
          Daniel Dai
        5. HIVE-16520.3.patch
          106 kB
          Daniel Dai
        6. HIVE-16520.4.patch
          107 kB
          Daniel Dai

          Issue Links

            Activity

              People

              • Assignee:
                daijy Daniel Dai
                Reporter:
                daijy Daniel Dai
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: