Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4206

Improve handling of invalid credentials to mitigate DOS issue (especially on SSL listeners)

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Do
    • Affects Version/s: 0.10.0.0, 0.10.0.1
    • Fix Version/s: None
    • Component/s: network, security
    • Labels:
      None

      Description

      The current handling of invalid credentials (ie wrong user/password) is to let the SaslException thrown from an implementation of javax.security.sasl.SaslServer.evaluateResponse()
      bubble up the call stack until it gets caught in
      org.apache.kafka.common.network.Selector.pollSelectionKeys()
      where the KafkaChannel gets closed - which will cause the client that made the request to be disconnected.

      This will happen however after the server has used considerable resources, especially for the SSL handshake which appears to be computationally expensive in Java.

      We have observed that if just a few clients keep repeating requests with the wrong credentials, it is quite easy to get all the network processing threads in the Kafka server busy doing SSL handshakes.

      This makes a Kafka cluster to easily suffer from a Denial Of Service - also non intentional - attack.
      It can be non intentional, i.e. also caused by friendly clients, for example because a Kafka Java client Producer supplied with the wrong credentials will not throw an exception on publishing, so it may keep attempting to connect without the caller realising.

      An easy fix which we have implemented and will supply a PR for is to delay considerably closing the KafkaChannel in the Selector, but obviously without blocking the processing thread.

      This has been tested to be very effective in reducing the cpu usage spikes caused by non malicious clients using invalid SASL PLAIN credentials over SSL.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ecomar Edoardo Comar
                Reporter:
                ecomar Edoardo Comar
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: