[KUDU-2905] Impala queries failed when master's IPKI CA info changed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 1.11.0
• Fix Version/s: n/a
• Component/s: client, master, security
• Assignee: Unassigned
• Reporter: Adar Dembo
• Created: 25/Jul/19 23:56
• Updated: 22/Jun/23 23:05
• Resolved: 22/Jun/23 23:05

Description

Saw this in a user report.

The cluster in question lost its master and the state was rebuilt from that of the tservers (see KUDU-2902). In doing so, the master lost its IPKI cert info and a new record was generated.

After the Kudu cluster was operational, the existing Impala cluster could not issue queries. All queries failed with an error like this:

Unable to open Kudu table: Runtime error: Client connection negotiation failed: client connection to 10.38.202.4:7051: TLS Handshake error: error:04067084:rsa routines:RSA_EAY_PUBLIC_DECRYPT:data too large for modulus:rsa_eay.c:738 error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:249 error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:s3_clnt.c:1264

Restarting Impala fixed it.

I'm not sure if this is an issue with how Impala caches KuduClient instances, or if it's an issue with how the client itself caches the master's CA certificate. For now I'm assuming this is a Kudu issue and the client needs to detect this error and invalidate existing certificates.

Issue Links

• relates to: KUDU-2661 Add a tool to reconstruct master data from the tablet servers