[IMPALA-7172] Statestore should verify that all subscribers are running the same version of Impala

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.13.0
• Component/s: Distributed Exec
• Labels: supportability
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 14/Jun/18 18:19
• Updated: 21/Dec/20 19:31

Description

While running a metadata test which uses sync_ddl=1, tests appeared to hang indefinitely.
Turns out one of the Impala daemons was running an older build which caused statestore topic updates to continuously fail.

Ideally the SS should track the version across subscribers and black list the ones that don't match the SS and CS version.

Logs from SS

I0614 11:11:04.410529 57312 statestore.cc:259] Preparing initial impala-membership topic update for [e-mail]:22000. Size = 2.06 KB
I0614 11:11:04.411222 57312 client-cache.cc:82] ReopenClient(): re-creating client for vb0204.halxg.cloudera.com:23000
I0614 11:11:04.411821 57312 client-cache.h:304] RPC Error: Client for vb0204.halxg.cloudera.com:23000 hit an unexpected exception: No more data to read., type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala20TUpdateStateResponseE, send: done
I0614 11:11:04.411831 57312 client-cache.cc:174] Broken Connection, destroy client for vb0204.halxg.cloudera.com:23000
I0614 11:11:04.411861 57312 statestore.cc:891] Unable to send priority topic update message to subscriber [e-mail]:22000, received error: RPC Error: Client for vb0204.halxg.cloudera.com:23000 hit an unexpected exception: No more data to read., type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala20TUpdateStateResponseE, send: done

Log from Impalad

I0614 11:03:19.479164 41915 thrift-util.cc:123] TAcceptQueueServer exception: N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data
I0614 11:03:19.680028 41916 thrift-util.cc:123] TAcceptQueueServer exception: N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data
I0614 11:03:19.680776 41917 thrift-util.cc:123] TAcceptQueueServer exception: N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data
I0614 11:03:19.881295 41918 thrift-util.cc:123] TAcceptQueueServer exception: N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data