[KAFKA-13427] Max connections are not blocked

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Ekaterina Chernousova
• Created: 02/Nov/21 00:24
• Updated: 09/Nov/21 09:54

Description

UPDATED 05/11/2021

According to the documentation, new connections should be blocked when the max connections limit is reached except it's an inter-broker listener http://kafka.apache.org/documentation/#brokerconfigs_max.connections 

Shouldn't connections to external listeners be blocked/dropped when max.connections limit is reached instead of waiting for an available connection slot? Thanks! 

 

The following tests have been executed:

Test 1: Broker-wide limit

Comment: connections to external listeners are not being blocked/dropped the same way as it happens when per.ip broker-wide limit is reached, instead the system waits for an available connection slot.

configuration

max.connection.creation.rate = 2147483647
max.connections = 10
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides = 

client logs when limit is reached (bin/kafka-console-consumer.sh)

[2021-11-05 05:56:57,714] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:57:30,915] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:58:01,022] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:58:10,899] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

 broker

2021-11-05 05:51:10,798 INFO [GroupCoordinator 0]: Assignment received from leader __strimzi-topic-operator-kstreams-45caeee4-1322-411d-a7f2-0bdcb9e38863-StreamThread-1-consumer-b6659f01-7981-4090-a9d3-89b9b80c340f for group __strimzi-topic-operator-kstreams for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-7]
2021-11-05 05:56:09,169 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 05:56:09,169 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 05:56:09,173 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 05:56:09,176 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]

Test 2: per.ip broker-wide limit

Comment: works as expected, however, it also blocks inter-broker connections

configuration

max.connection.creation.rate = 2147483647
max.connections = 100
max.connections.per.ip = 10
max.connections.per.ip.overrides = 

client logs when limit is reached (bin/kafka-console-consumer.sh)

[2021-11-05 06:43:06,832] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:06,937] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,042] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,144] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,246] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

 broker

2021-11-05 06:43:06,728 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:06,831 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:06,934 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,039 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,142 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,246 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,349 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]

Test 3: listener-level limit

Comment: I assume connections should be blocked/dropped instead of waiting for an available connection slot

configuration

max.connection.creation.rate = 2147483647
max.connections = 2147483647
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides = 

listener.name.example-9095.max.connections=10

client logs when limit is reached (bin/kafka-console-consumer.sh)

[2021-11-05 06:38:05,084] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:38:38,175] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:39:08,278] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:39:17,228] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

 broker

2021-11-05 06:36:22,266 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:36:22,267 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:36:22,274 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:36:22,274 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]

Test 4: Broker-wide limit with inter-broker connection

Comment: works as expected

configuration

max.connection.creation.rate = 2147483647
max.connections = 10
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides = 

client that connects to an internal listener logs when limit is reached (bin/kafka-console-consumer.sh)

[2021-11-05 06:56:51,602] WARN [Consumer clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479] Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:57:24,697] WARN [Consumer clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479] Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:57:57,797] WARN [Consumer clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479] Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

 broker

2021-11-05 06:52:42,869 INFO [GroupCoordinator 0]: Assignment received from leader __strimzi-topic-operator-kstreams-ece4c6be-1043-467c-8785-872910af313f-StreamThread-1-consumer-1ed22ca1-b161-40ab-921f-994cdabe5f82 for group __strimzi-topic-operator-kstreams for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-4]
2021-11-05 06:57:40,509 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:57:40,509 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:57:40,518 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:57:40,518 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]