Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2096

Enable keepalive socket option for broker to prevent socket leak

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.1
    • Fix Version/s: 0.9.0.0
    • Component/s: network
    • Labels:
      None

      Description

      We run a Kafka 0.8.2.1 cluster in AWS with large number of producers (> 10000). Also the number of producer instances scale up and down significantly on a daily basis.

      The issue we found is that after 10 days, the open file descriptor count will approach the limit of 32K. An investigation of these open file descriptors shows that a significant portion of these are from client instances that are terminated during scaling down. Somehow they still show as "ESTABLISHED" in netstat. We suspect that the AWS firewall between the client and broker causes this issue.

      We attempted to use "keepalive" socket option to reduce this socket leak on broker and it appears to be working. Specifically, we added this line to kafka.network.Acceptor.accept():

      socketChannel.socket().setKeepAlive(true)

      It is confirmed during our experiment of this change that entries in netstat where the client instance is terminated were probed as configured in operating system. After configured number of probes, the OS determined that the peer is no longer alive and the entry is removed, possibly after an error in Kafka to read from the channel and closing the channel. Also, our experiment shows that after a few days, the instance was able to keep a stable low point of open file descriptor count, compared with other instances where the low point keeps increasing day to day.

        Attachments

        1. patch.diff
          0.7 kB
          Allen Wang

          Activity

            People

            • Assignee:
              allenxwang Allen Wang
              Reporter:
              allenxwang Allen Wang
              Reviewer:
              Jun Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: