[KAFKA-3328] SimpleAclAuthorizer can lose ACLs with frequent add/remove calls

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.10.0.0
• Assignee: Grant Henke
• Reporter: Grant Henke
• Reviewer: Flavio Paiva Junqueira
• Created: 03/Mar/16 17:27
• Updated: 20/Mar/16 07:46
• Resolved: 20/Mar/16 07:46

Description

Currently when adding or removing an ACL with the SimpleAclAuthorizer the following high level steps happen:

1. read acls from cache
2. merge with the changes acls
3. update zookeeper
4. add a change notification

Then the Authorizers listening for the change notification know to invalidate their cache and get the latest value. However that takes some time. In the time between the ACL change and the cache update, a new add or remove request could be made. This will follow the steps listed above, and if the cache is not correct all changes from the previous request are lost.

This can be solved on a single node, by updating the cache at the same time you update zookeeper any time a change is made. However, because there can be multiple instances of the Authorizer, a request could come to a separate authorizer and overwrite the Zookeeper state again loosing changes from earlier requests.

To solve this on multiple instances. The authorizer could always read/write state from zookeeper (instead of the cache) for add/remove requests and only leverage the cache for get/authorize requests. Or it could block until all the live instances have updated their cache.

Below is a log from a failed test in the WIP pull request for KAFKA-3266 that shows this behavior:

[2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow permission for operations: Describe from hosts: * for Cluster:kafka-cluster (kafka.server.KafkaApis:52)
[2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny permission for operations: Describe from hosts: * for Cluster:kafka-cluster (kafka.server.KafkaApis:52)
[2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)

Issue Links

• breaks: KAFKA-3266 Implement KIP-140 RPCs and APIs for creating, altering, and listing ACLs
• is blocked by: KAFKA-3403 Upgrade ZkClient to 0.8