[KUDU-2265] A non-leader master uses self-signed server TLS cert if it hasn't ever run as a leader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.6.0
• Fix Version/s: 1.7.0
• Component/s: master, security
• Target Version/s: 1.7.0
• Code Review: http://gerrit.cloudera.org:8080/9076
• Assignee: Alexey Serbin
• Reporter: Alexey Serbin
• Created: 18/Jan/18 19:55
• Updated: 21/Feb/18 01:53
• Resolved: 26/Jan/18 01:28

Description

As it's currently implemented, master process replaces its auto-generated self-signed server TLS certificate with CA-signed one only when it becomes a leader (see CatalogManager::InitCertAuthorityWith() method; it's invoked only from CatalogManager::InitCertAuthority(), that is invoked only from CatalogManager::PrepareForLeadershipTask()).

 

In case of just one Raft election from the start (which is pretty common, BTW), non-leader masters run without CA-signed server certificate for a long time in case of a multi-master configuration.  That causes clients to not use their authn tokens for authentication while connecting to those non-leader masters.  In case of Spark applications where executors do not have Kerberos credentials (the common case), application logs are polluted with messages like below:

org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (kinit)
  at org.apache.kudu.client.Negotiator.evaluateChallenge(Negotiator.java:705)
  at org.apache.kudu.client.Negotiator.sendSaslInitiate(Negotiator.java:581)

...

Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
  at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

Issue Links

• is related to: KUDU-2267 Client may see negotiation failure when talks to master followers with only self signed cert
• relates to: KUDU-2319 Follower masters cannot accept authn tokens for verification