[KAFKA-13461] KafkaController stops functioning as active controller after ZooKeeperClient auth failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0, 3.0.1
• Component/s: zkclient
• Assignee: Rajini Sivaram
• Reporter: Vincent Jiang
• Reviewer: Jun Rao
• Created: 17/Nov/21 20:25
• Updated: 24/Feb/22 22:13
• Resolved: 02/Dec/21 22:27

Description

When java.security.auth.login.config is present, but there is no "Client" section,  ZookeeperSaslClient creation fails and raises LoginExcpetion, result in warning log:

WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '***'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

When this happens after initial startup, ClientCnxn enqueues an AuthFailed event which will trigger following sequence:

1. zkclient reinitialization is triggered
2. the controller resigns.
3. Before the controller's ZK session expires, the controller successfully connect to ZK and maintains the current session
4. In KafkaController.elect(), the controller sets activeControllerId to itself and short-circuits the rest of the elect. Since the controller resigned earlier and also skips the call to onControllerFailover(), the controller is not actually functioning as the active controller (e.g. the necessary ZK watchers haven't been registered).

 

Issue Links

• fixes: KAFKA-13407 Kafka controller out of service after ZK leader restart
• links to: GitHub Pull Request #11563