[IMPALA-12286] Make leader election robust for Impala CatalogD HA

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.3.0
• Fix Version/s: Impala 4.3.0
• Component/s: Backend
• Epic Color: ghx-label-13
• Assignee: Wenzhe Zhou
• Reporter: Wenzhe Zhou
• Due: 28/Jul/23
• Created: 14/Jul/23 08:55
• Updated: 26/Jul/23 07:44
• Resolved: 21/Jul/23 19:25

Description

statestore is responsible for electing a leader catalogd when CatalogD HA is enabled for an Impala cluster with a pair of catalogd in active-passive mode. When a new leader catalogd is elected, statestore notify all coordinators and catalogds for the new leader catalogd in thrift RPCs. But RPCs could be failed due to network condition. We have to resend RPCs to the coordinators and catalogds if they are still in the cluster. Otherwise, catalogds don't know their new roles, and the coordinators still access old leader catalogd.
We also need to rename metric name ""catalog-server.ha-active-status" to "catalog-server.active-status".