Kafka
  1. Kafka
  2. KAFKA-1041

Number of file handles increases indefinitely in producer if broker host is unresolvable

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.2.0
    • Component/s: producer
    • Labels:
    • Environment:
      *unix*

      Description

      We found a issue that if broker host is un resolvable, the number of file handle keep increasing for every message we produce and eventually it uses up all available files handles in operating system. If broker itself is not running and broker host name is resolvable, open file handles count stays flat.

      lsof output shows number of these open file handles continue to grow for every message we produce.

      java 19631 relango 81u sock 0,6 0t0 196966526 can't identify protocol

      I can easily reproduce this with console producer, If I run console producer with right hostname and if broker is not running, the console producer will exit after three tries. But If I run console producer with unresolvable broker, it throws below exception and continues to wait for user input, every time I enter new message, it opens socket and file handle count keeps increasing..

      Here is Exception in producer

      ERROR fetching topic metadata for topics [Set(test-1378245487417)] from broker [ArrayBuffer(id:0,host:localhost1,port:6667)] failed (kafka.utils.Utils$)
      kafka.common.KafkaException: fetching topic metadata for topics [Set(test-1378245487417)] from broker [ArrayBuffer(id:0,host:localhost1,port:6667)] failed
      at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:51)
      at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
      at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:79)
      at kafka.utils.Utils$.swallow(Utils.scala:186)
      at kafka.utils.Logging$class.swallowError(Logging.scala:105)
      at kafka.utils.Utils$.swallowError(Utils.scala:45)
      at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:79)
      at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
      at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
      at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
      at scala.collection.immutable.Stream.foreach(Stream.scala:526)
      at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
      at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
      Caused by: java.nio.channels.UnresolvedAddressException
      at sun.nio.ch.Net.checkAddress(Net.java:30)
      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:487)
      at kafka.network.BlockingChannel.connect(BlockingChannel.scala:59)
      at kafka.producer.SyncProducer.connect(SyncProducer.scala:151)
      at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:166)
      at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:73)
      at kafka.producer.SyncProducer.send(SyncProducer.scala:117)
      at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:37)
      ... 12 more

        Issue Links

          Activity

          Rajasekar Elango created issue -
          Neha Narkhede made changes -
          Field Original Value New Value
          Assignee Jun Rao [ junrao ] Neha Narkhede [ nehanarkhede ]
          Hide
          Jun Rao added a comment -

          Moving to 0.8.1 since it's not very critical

          Show
          Jun Rao added a comment - Moving to 0.8.1 since it's not very critical
          Jun Rao made changes -
          Fix Version/s 0.8.1 [ 12322960 ]
          Neha Narkhede made changes -
          Fix Version/s 0.8.2 [ 12326167 ]
          Fix Version/s 0.8.1 [ 12322960 ]
          Neha Narkhede made changes -
          Labels features features newbie
          Anish Khanzode made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Anish Khanzode added a comment -

          Is this good enough for this?

          Show
          Anish Khanzode added a comment - Is this good enough for this?
          Anish Khanzode made changes -
          Attachment KAFKA-1041-patch.diff [ 12631545 ]
          Hide
          Jun Rao added a comment -

          Thanks for the patch. Some comments.

          1. We probably should catch all throwables.
          2. We can probably just call disconnect() when we hit an exception.

          Show
          Jun Rao added a comment - Thanks for the patch. Some comments. 1. We probably should catch all throwables. 2. We can probably just call disconnect() when we hit an exception.
          Anish Khanzode made changes -
          Attachment KAFKA-1041-patch.diff [ 12631545 ]
          Hide
          Anish Khanzode added a comment - - edited

          Here is the updated patch for 0.8
          Thanks for looking into it.

          Show
          Anish Khanzode added a comment - - edited Here is the updated patch for 0.8 Thanks for looking into it.
          Anish Khanzode made changes -
          Attachment KAFKA-1041-patch.diff [ 12631759 ]
          Hide
          Jun Rao added a comment -

          Thanks for patch v2. +1 and committed to trunk.

          Show
          Jun Rao added a comment - Thanks for patch v2. +1 and committed to trunk.
          Jun Rao made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Assignee Neha Narkhede [ nehanarkhede ] Rajasekar Elango [ erajasekar ]
          Resolution Fixed [ 1 ]
          Hide
          Anish Khanzode added a comment -

          Is 0.8.1 released? Can I get this applied on a released stable branch?

          Show
          Anish Khanzode added a comment - Is 0.8.1 released? Can I get this applied on a released stable branch?
          Hide
          Jun Rao added a comment -

          0.8.1 is being voted now.

          Show
          Jun Rao added a comment - 0.8.1 is being voted now.
          Neha Narkhede made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Chris Riccomini made changes -
          Link This issue is required by SAMZA-101 [ SAMZA-101 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          176d 44m 1 Anish Khanzode 27/Feb/14 15:28
          Patch Available Patch Available Resolved Resolved
          1d 35m 1 Jun Rao 28/Feb/14 16:03
          Resolved Resolved Closed Closed
          110d 13h 8m 1 Neha Narkhede 19/Jun/14 06:12

            People

            • Assignee:
              Rajasekar Elango
              Reporter:
              Rajasekar Elango
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development