[IMPALA-12326] Impala daemons should only subscribe to statestore once rpc services are ready

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Backend
• Epic Color: ghx-label-8
• Assignee: Riza Suminto
• Reporter: Abhishek Rawat
• Created: 31/Jul/23 21:37
• Updated: 28/Aug/23 13:01
• Resolved: 02/Aug/23 02:43

Description

The Impala daemons start the statestore subscriber service before the krpc/rpc services are ready:

https://github.com/apache/impala/blob/branch-4.2.0/be/src/service/impala-server.cc#L2934

As a result, there is a small window where statestore could try to connect with Impala daemons, but the rpc service isn't ready and so statestore logs get filled with thrift timeout errors:

RPC Error: Client for 10.80.205.184:23000 hit an unexpected exception: No more data to read., type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala18THeartbe
I0731 19:43:09.058470    79 client-cache.cc:174] Broken Connection, destroy client for 10.80.205.184:23000
I0731 19:43:09.076826    83 client-cache.h:362] RPC Error: Client for 10.80.192.41:23000 hit an unexpected exception: No more data to read., type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala18THeartbea 

It makes sense for statestore subscriber on Impala daemons to only start once the rpc/krpc service has started successfully.