[KAFKA-3783] Race condition on last ACL removal for a resource fails with a ZkBadVersionException

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.9.0.1, 0.10.0.0
• Fix Version/s: 0.10.0.1
• Flags: Patch
• Assignee: Sébastien Launay
• Reporter: Sébastien Launay
• Created: 02/Jun/16 21:01
• Updated: 07/Jun/16 00:41
• Resolved: 07/Jun/16 00:41

Description

When removing the last ACL for a given resource, the znode storing the ACLs will get removed.
The version number of the znode is used for optimistic locking in a loop to provide atomic changes across brokers.

Unfortunately the exception thrown when the operation fails because of a different version number is the wrong one (KeeperException.BadVersionException instead of ZkClient ZkBadVersionException) and does not get caught resulting in the following stack trace:

org.I0Itec.zkclient.exception.ZkBadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /kafka-acl/Topic/e6df8028-f268-408c-814e-d418e943b2fa
	at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:51)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:1000)
	at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:1047)
	at kafka.utils.ZkUtils.conditionalDeletePath(ZkUtils.scala:522)
	at kafka.security.auth.SimpleAclAuthorizer.kafka$security$auth$SimpleAclAuthorizer$$updateResourceAcls(SimpleAclAuthorizer.scala:282)
	at kafka.security.auth.SimpleAclAuthorizer$$anonfun$removeAcls$1.apply$mcZ$sp(SimpleAclAuthorizer.scala:187)
	at kafka.security.auth.SimpleAclAuthorizer$$anonfun$removeAcls$1.apply(SimpleAclAuthorizer.scala:187)
	at kafka.security.auth.SimpleAclAuthorizer$$anonfun$removeAcls$1.apply(SimpleAclAuthorizer.scala:187)
	at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
	at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
	at kafka.security.auth.SimpleAclAuthorizer.removeAcls(SimpleAclAuthorizer.scala:186)
	...
Caused by: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /kafka-acl/Topic/e6df8028-f268-408c-814e-d418e943b2fa
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
	at org.I0Itec.zkclient.ZkConnection.delete(ZkConnection.java:109)
	at org.I0Itec.zkclient.ZkClient$11.call(ZkClient.java:1051)
	at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:990)
	... 18 more

I noticed this behaviour while working on another fix when running the SimpleAclAuthorizerTest unit tests but this can happens when running simultaneously the kafka-acls.sh command on different brokers in rare cases.

Issue Links

• links to: GitHub Pull Request #1461