[KUDU-2580] C++ client does not re-acquire authn token in some scenarios in case of multi-master Kudu cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.7.1
• Fix Version/s: 1.8.0
• Component/s: client, security
• Code Review: http://gerrit.cloudera.org:8080/11449
• Assignee: Alexey Serbin
• Reporter: Alexey Serbin
• Created: 17/Sep/18 04:10
• Updated: 26/Sep/18 22:46
• Resolved: 19/Sep/18 21:08

Description

In case of multi-master Kudu cluster, the C++ client fails to re-acquire authn token in the following scenario:

1. Client is running against a multi-master cluster.
2. Client successfully authenticates and gets an authn token by calling ConnectToCluster.
3. Client keeps the connection to leader master open, but follower masters close connections to the client due to inactivity.
4. After the authn token expires, a change in the master leadership happens.
5. Client tries to open a table, first making a request to the former leader master. However, the former leader returns NOT_THE_LEADER error.

Eventually, the latter operation fails out with the following error:
Timed out: GetTableSchema timed out after deadline expired

In the case of that error, along with the message above, the client is also logging messages like:

Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to IP:port: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired

Issue Links

• relates to: KUDU-2586 Refactor ConnectToClusterRpc to handle FATAL_INVALID_AUTHENTICATION_TOKEN