[KAFKA-9806] Unable to consume when __consumer_offsets is created under insufficient cluster ACLs

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consumer, controller, core
• Assignee: Paolo Moriello
• Reporter: Paolo Moriello
• Created: 02/Apr/20 14:37
• Updated: 09/Apr/20 15:48

Description

Description

The __consumer_offsets topic is automatically created the first time consumption happens on a cluster. When a consumer start to consume from a topic, it first submits a FindCoordinator request, which tries to get metadata from the __consumer_offsets topic. If this does not exist, it gets created. On topic creation, the controller assigning partitions to the other brokers through UpdateMetadata and LeaderAndIsr requests.

If there aren't sufficient cluster-level ACLs, UpdateMetadata and LeaderAndIsr are not authorized by peer brokers; so, the requests fail, but the auto-creation of the topic succeeds and zk-nodes are filled with the related metadata. No-operation is performed by the controller on handling of failed UpdateMetadata and LeaderAndIsr, so __consumer_offset topic does not get deleted.

When ACLs are removed/fixed, __consumer_offset is in a bad state for the topics we tried to consume on under insufficient ACLs, and consumption is not possible: consumer does not return any data. No errors are thrown by the consumer, nor by the brokers. By enabling DEBUG logs on the consumer we only see that: 'The coordinator is not available.'

Note: this is fixed with a broker restart or __consumer_offsets partition reassignment, since both operations trigger UpdateMetadata and LeaderAndIsr requests on the topic.

Steps to reproduce

Clean Kafka cluster (no __consumer_offset) with more than 1 broker, and SSL enabled for inter-broker communication.

+ I had allow.everyone.if.no.acl.found set to true on my test cluster.

1. Create a topic
2. Set insufficient cluster level ACLs. Which precise ACL does not matter as long as ClusterAction initiated by administrative messages between brokers is denied. 
3. Consume on the topic -> this request will fail with unauthorized errors on the client side as well
4. Remove ACLs or set ACLs to allow inter-broker communication
5. Produce/consume on topic. Consumer won't be able to read any data.

Step 3 triggers creation of __consumer_offset topic. In addition to the client side errors, it is possible to see related UpdateMetadata and LeaderAndIsr exceptions in the broker logs. However at this point you can verify that __consumer_offset topic is created.

 

 

Issue Links

• links to: GitHub Pull Request #8415