Minor Description
The Connector::taskConfigs(int maxTasks) function is used by Connectors to enumerate tasks configurations. This takes an argument which comes from the tasks.max connector config. This is the Javadoc for that method:
/**
* Returns a set of configurations for Tasks based on the current configuration,
* producing at most {@code maxTasks} configurations.
*
* @param maxTasks maximum number of configurations to generate
* @return configurations for Tasks
*/
public abstract List<Map<String, String>> taskConfigs(int maxTasks);
This includes the constraint that the number of tasks is at most maxTasks, but this constraint is not enforced by the framework.
To enforce this constraint, we could begin dropping configs that exceed the limit, and log a warning. For sink connectors this should harmlessly rebalance the consumer subscriptions onto the remaining tasks. For source connectors that distribute their work via task configs, this may result in an interruption in data transfer.
Attachments
Issue Links
- is related to
-
KAFKA-15559 KIP-987: Connect Static Assignments
-
- Open
-
- links to
- mentioned in
-
Page Loading...