[KAFKA-13244] Control Which Brokers Host Partitions of Newly Created Topics

• Type: New Feature
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: admin
• Assignee: Unassigned
• Reporter: Michael Jaschob
• Created: 27/Aug/21 17:34
• Updated: 18/Oct/21 08:31

Description

When new topics are created through the admin interface, the Kafka controller creates the partition assignments for these topics according to the algorithm defined in AdminUtils.assignReplicasToBrokers. All (alive) brokers in the cluster become candidates for hosting partitions.

However, sometimes cluster administrators don't want certain brokers to host partitions for newly created topics. This can be for a variety of reasons, e.g.:

• a broker may be on a host that is slated for retirement and about to be shut down
• a broker may be acting as a canary of a new Kafka version and is being tested with a controlled set of topic-partitions, and should be restricted from hosting any other topics
• similarly, a broker may in use for administrative operations, such as dumping log segments, and consequently may be unstable and ideally as little-used as possible

One solution is for the cluster admins to not give out CREATE permissions to clients, and instead provide a service/abstraction for creating new topics with explicit partition assignments. Unfortunately, in a world with Kafka Streams applications, this is too restrictive: Kafka Streams clients expect to use the Kafka tooling to create topics when their topology changes, or when they need to reset their apps.

Since there is currently there is no way to control which brokers are eligible to be assigned newly created partitions, I am proposing a new configuration parameter, create.topic.broker.filter.policy.class.name to allow cluster administrators to provide a pluggable class to control whether a broker should be allowed to host new partitions. The class implements a simple interface to give a binary yes/no verdict on each broker:

boolean isAllowedToHostPartitions(Broker broker);

This follows a pattern similar to create.topic.policy.class.name and alter.topic.policy.class.name, which also provide cluster-level control over other aspects of topic creation.

I have a PR ready which I'll submit after creating this ticket. I'm not sure if this feature idea needs a KIP - technically, it's a change to configuration, but on the other hand it's a pretty small change that aligns closely to existing functionality (create/alter topic policies). Please let me know if I should follow the KIP process.

Issue Links

• links to: GitHub Pull Request #11277