[IMPALA-11221] Impala loses to trace Kudu leader balance

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 3.4.0
• Component/s: Catalog
• Labels: high-availability impala kudu
• Environment: Red Hat Enterprise Linux Server 7.8 ; CDP 7.1.6 ; CM 7.1.3
• Flags: Important
• Target Version: Impala 3.4.0
• Epic Color: ghx-label-14
• Assignee: Unassigned
• Reporter: Juanes Palma Bueno
• Created: 04/Apr/22 10:34
• Updated: 04/Apr/22 10:34

Description

In an HA cluster with 3 Kudu Master servers, (see attachment)

when the server where Kudu leader goes down (Master2) all queries against kudu become unavailable.

Query: select * from  foo_table limit 3
Query submitted at: 2022-04-04 08:38:32 (Coordinator: http://worker2.server.com:25000)
ERROR: ImpalaRuntimeException: Unable to initialize the Kudu scan node
CAUSED BY: AnalysisException: Unable to open the Kudu table: schema.foo_table
CAUSED BY: NonRecoverableException: cannot complete before timeout: KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=1, TimeoutTracker(timeout=180000, elapsed=180004), Trace Summary(0 ms): Sent(1), Received(0), Delayed(0), MasterRefresh(0), AuthRefresh(0), Truncated: false
 Sent: (master2:7051, [ GetTableSchema, 1 ]))

The coordinator it's trying to get the metadata information from the Kudu Master that is already down (master2), meanwhile there are still 2 Kudu Masters UP and one of them was already elected as Leader:

               uuid               |          rpc-addresses          |   role
-------------------------------------------------------------------------
 daa9129e78244be2aaa7e5e649cc1dc8 | master1:7051 | FOLLOWER
 1f02c618009c44d381c55841dcb5a498 | master3:7051 | LEADER