[HIVE-27692] Explore removing the always task from embedded HMS

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Standalone Metastore
• Labels: pull-request-available
• Assignee: Zhihua Deng
• Reporter: Zhihua Deng
• Created: 15/Sep/23 03:52
• Updated: 29/Mar/24 17:50
• Resolved: 23/Feb/24 09:12

Description

The always tasks are running in the leader HMS now, the properties for configuring the leader should only belong to HMS, other engines such as Spark/Impala doesn't need to know these properties. For most cases, the engine only cares about the properties for connecting HMS, e.g, hive.metastore.uris.

Every time when a new apps uses an embedded Metastore, it will start the HMS always tasks by default. Imaging we have hundreds of apps, then hundreds of pieces of the same tasks are running, this will put extra burden to the underlying databases, such as the flooding queries, connection limit.

I think we can remove always tasks from the embeded Metastore, the always task will be taken care of by the standalone Metastore, as a standalone Metastore should be here in production environment.

Issue Links

• links to: GitHub Pull Request #4782