Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1017

High number of open file handles in 0.8 producer

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: producer
    • Labels:
      None

      Description

      Reported by Jun Rao:

      For over-partitioned topics, each broker could be the leader for at least 1 partition. In the producer, we randomly select a partition to send the data. Pretty soon, each producer will establish a connection to each of the n brokers. Effectively, we increased the # of socket connections by a factor of n, compared to 0.7.

      The increased number of socket connections increases the number of open file handles, this could come pretty close to the OS limit.

        Attachments

          Activity

            People

            • Assignee:
              swapnilghike Swapnil Ghike
              Reporter:
              swapnilghike Swapnil Ghike
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: