[HIVE-26509] Introduce dynamic leader election in HMS

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Component/s: Standalone Metastore
• Labels: pull-request-available
• Assignee: Zhihua Deng
• Reporter: Zhihua Deng
• Created: 01/Sep/22 06:34
• Updated: 15/Aug/23 06:33
• Resolved: 14/Dec/22 01:09

Description

From HIVE-21841 we have a leader HMS selected by configuring metastore.housekeeping.leader.hostname on startup. This approach saves us from running duplicated HMS's housekeeping tasks cluster-wide. 

In this jira, we introduce another dynamic leader election: adopt hive lock to implement the leader election. Once a HMS owns the lock, then it becomes the leader, carries out the housekeeping tasks, and sends heartbeats to renew the lock before timeout. If the leader fails to reclaim the lock, then stops the already started tasks if it has, the electing event is audited. We can achieve a more dynamic leader when the original goes down or in the public cloud without well configured property, and reduce the leader’s burdens by running these tasks among different leaders.

Issue Links

• is related to: HIVE-27438 Audit leader election event failed in non-appendable filesystems
• links to: GitHub Pull Request #3567