Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13427

Max connections are not blocked

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.0.0
    • None
    • core
    • None

    Description

      UPDATED 05/11/2021

      According to the documentation, new connections should be blocked when the max connections limit is reached except it's an inter-broker listener http://kafka.apache.org/documentation/#brokerconfigs_max.connections 

      Shouldn't connections to external listeners be blocked/dropped when max.connections limit is reached instead of waiting for an available connection slot? Thanks! 

       

      The following tests have been executed:

      Test 1: Broker-wide limit

      Comment: connections to external listeners are not being blocked/dropped the same way as it happens when per.ip broker-wide limit is reached, instead the system waits for an available connection slot.

      configuration

      max.connection.creation.rate = 2147483647
      max.connections = 10
      max.connections.per.ip = 2147483647
      max.connections.per.ip.overrides = 
      

      client logs when limit is reached (bin/kafka-console-consumer.sh)

      [2021-11-05 05:56:57,714] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 05:57:30,915] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 05:58:01,022] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 05:58:10,899] WARN [Consumer clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553] Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      

       broker

      2021-11-05 05:51:10,798 INFO [GroupCoordinator 0]: Assignment received from leader __strimzi-topic-operator-kstreams-45caeee4-1322-411d-a7f2-0bdcb9e38863-StreamThread-1-consumer-b6659f01-7981-4090-a9d3-89b9b80c340f for group __strimzi-topic-operator-kstreams for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-7]
      2021-11-05 05:56:09,169 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 05:56:09,169 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 05:56:09,173 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 05:56:09,176 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
      

      Test 2: per.ip broker-wide limit

      Comment: works as expected, however, it also blocks inter-broker connections

      configuration

      max.connection.creation.rate = 2147483647
      max.connections = 100
      max.connections.per.ip = 10
      max.connections.per.ip.overrides = 
      

      client logs when limit is reached (bin/kafka-console-consumer.sh)

      [2021-11-05 06:43:06,832] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:43:06,937] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:43:07,042] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:43:07,144] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:43:07,246] WARN [Consumer clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      

       broker

      2021-11-05 06:43:06,728 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
      2021-11-05 06:43:06,831 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
      2021-11-05 06:43:06,934 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
      2021-11-05 06:43:07,039 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
      2021-11-05 06:43:07,142 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
      2021-11-05 06:43:07,246 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
      2021-11-05 06:43:07,349 INFO Rejected connection from /172.17.0.1, address already has the configured maximum of 10 connections. (kafka.network.Acceptor) [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
      

      Test 3: listener-level limit

      Comment: I assume connections should be blocked/dropped instead of waiting for an available connection slot

      configuration

      max.connection.creation.rate = 2147483647
      max.connections = 2147483647
      max.connections.per.ip = 2147483647
      max.connections.per.ip.overrides = 
      
      listener.name.example-9095.max.connections=10
      

      client logs when limit is reached (bin/kafka-console-consumer.sh)

      [2021-11-05 06:38:05,084] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:38:38,175] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:39:08,278] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:39:17,228] WARN [Consumer clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043] Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      

       broker

      2021-11-05 06:36:22,266 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 06:36:22,267 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 06:36:22,274 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 06:36:22,274 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
      

      Test 4: Broker-wide limit with inter-broker connection

      Comment: works as expected

      configuration

      max.connection.creation.rate = 2147483647
      max.connections = 10
      max.connections.per.ip = 2147483647
      max.connections.per.ip.overrides = 
      

      client that connects to an internal listener logs when limit is reached (bin/kafka-console-consumer.sh)

      [2021-11-05 06:56:51,602] WARN [Consumer clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479] Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:57:24,697] WARN [Consumer clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479] Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      [2021-11-05 06:57:57,797] WARN [Consumer clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479] Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
      

       broker

      2021-11-05 06:52:42,869 INFO [GroupCoordinator 0]: Assignment received from leader __strimzi-topic-operator-kstreams-ece4c6be-1043-467c-8785-872910af313f-StreamThread-1-consumer-1ed22ca1-b161-40ab-921f-994cdabe5f82 for group __strimzi-topic-operator-kstreams for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-4]
      2021-11-05 06:57:40,509 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 06:57:40,509 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 06:57:40,518 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 HashMap() (kafka.controller.KafkaController) [controller-event-thread]
      2021-11-05 06:57:40,518 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            eachernous Ekaterina Chernousova
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: