[HIVE-23949] Introduce caching layer in HS2 to accelerate query compilation

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: HiveServer2
• Labels: pull-request-available
• Assignee: Soumyakanti Das
• Reporter: Soumyakanti Das
• Created: 29/Jul/20 20:04
• Updated: 27/Feb/24 22:24
• Resolved: 03/Aug/20 15:01

Description

One of the major contributors to compilation latency is the retrieval of metadata from HMS. This JIRA introduces a caching layer in HS2 for this metadata. Its design is simple, relying on snapshot information being queried to cache and invalidate the metadata. This will help us to reduce the time spent in compilation by using HS2 memory more effectively, and it will allow us to improve HMS throughput for multi-tenant workloads by reducing the number of calls it needs to serve.
This patch only caches partition retrieval information, which is often one of the most costly metadata operations. It also sets the foundation to cache additional calls in follow-up work.

Issue Links

• is related to: HIVE-25918 Invalid stats after multi inserting into the same partition
• is related to: HIVE-24034 Add getTable to HS2 local cache
• is related to: HIVE-24025 Add getAggrStatsFor to HS2 cache
• is related to: HIVE-24176 Create query-level cache for HMS requests and extend existing local HS2 HMS cache
• Parent Feature: HIVE-23979 Resolve spotbugs errors in JsonReporter.java, Metrics.java, and PerfLogger.java
• Parent Feature: HIVE-23978 Enable logging with PerfLogger in HMS client
• links to: GitHub Pull Request #1317