Kafka
  1. Kafka
  2. KAFKA-370

Exception "java.util.NoSuchElementException: None.get" appears inconsistently in Mirror Maker log.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:
      None

      Description

      Exception in Mirror Maker log:
      =========================
      [2012-06-20 10:56:04,364] DEBUG Getting broker partition info for topic test01 (kafka.producer.BrokerPartitionInfo)
      [2012-06-20 10:56:04,365] INFO Fetching metadata for topic test01 (kafka.producer.BrokerPartitionInfo)
      [2012-06-20 10:56:04,366] ERROR Error in handling batch of 200 events (kafka.producer.async.ProducerSendThread)
      java.util.NoSuchElementException: None.get
      at scala.None$.get(Option.scala:185)
      at scala.None$.get(Option.scala:183)
      at kafka.producer.ProducerPool.getAnyProducer(ProducerPool.scala:76)
      at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:73)
      at kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:45)
      at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:129)
      at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:95)
      at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:94)
      at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
      at scala.collection.immutable.List.foreach(List.scala:45)
      at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:44)
      at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:42)
      at kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:94)
      at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:65)
      at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:49)
      at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:96)
      at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:82)
      at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:60)
      at scala.collection.immutable.Stream.foreach(Stream.scala:254)
      at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:59)
      at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:37)

      Steps to reproduce
      =================
      It cannot be reproduced consistently. However, running the following script 2 or 3 times (step 2) will show the error:

      1. Apply kafka-306-v2.patch to 0.8 branch (revision 1352192 is used to reproduce this)

      2. Under the directory <kafka home>/system_test/broker_failure, execute the following command:
      => $ bin/run-test.sh 5 0

      3. Check the log under the directory <kafka home>/system_test/broker_failure:
      => $ grep Exception `ls kafka_mirror_maker*.log`
      => kafka_mirror_maker2.log:java.util.NoSuchElementException: None.get

      4. Also the kafka log sizes between source and target will not match:

      [/tmp] $ find kafka* -name *.kafka -ls
      19400444 6104 rw-rr- 1 jfung eng 6246655 Jun 20 10:56 kafka-source4-logs/test01-0/00000000000000000000.kafka
      19400819 5356 rw-rr- 1 jfung eng 5483627 Jun 20 10:56 kafka-target3-logs/test01-0/00000000000000000000.kafka

      Notes about the patch kafka-306-v2.patch
      ===============================
      This patch fix the broker_failure test suite to do the followings:

      a. Start 4 kafka brokers as source cluster
      b. Start 3 kafka brokers as target cluster
      c. Start 3 mirror maker to enable mirroring
      d. Send n messages to source cluster
      e. No bouncing is performed in this test for simplicity
      f. After the producer is stopped, validate the data count is matched between source & target

        Activity

        John Fung created issue -
        John Fung made changes -
        Field Original Value New Value
        Description ** Exception in Mirror Maker log:

        [2012-06-20 10:56:04,364] DEBUG Getting broker partition info for topic test01 (kafka.producer.BrokerPartitionInfo)
        [2012-06-20 10:56:04,365] INFO Fetching metadata for topic test01 (kafka.producer.BrokerPartitionInfo)
        [2012-06-20 10:56:04,366] ERROR Error in handling batch of 200 events (kafka.producer.async.ProducerSendThread)
        java.util.NoSuchElementException: None.get
                at scala.None$.get(Option.scala:185)
                at scala.None$.get(Option.scala:183)
                at kafka.producer.ProducerPool.getAnyProducer(ProducerPool.scala:76)
                at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:73)
                at kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:45)
                at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:129)
                at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:95)
                at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:94)
                at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
                at scala.collection.immutable.List.foreach(List.scala:45)
                at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:44)
                at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:42)
                at kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:94)
                at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:65)
                at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:49)
                at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:96)
                at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:82)
                at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:60)
                at scala.collection.immutable.Stream.foreach(Stream.scala:254)
                at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:59)
                at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:37)


        ** It cannot be reproduced consistently. However, running the following script 2 or 3 times will show the error:

        1. Apply kafka-306-v2.patch to 0.8 branch (revision 1352192 is used to reproduce this)

        2. Under the directory <kafka home>/system_test/broker_failure, execute the following command:
        => $ bin/run-test.sh 5 0

        3. Check the log under the directory <kafka home>/system_test/broker_failure:
        => $ grep Exception `ls kafka_mirror_maker*.log`
        => kafka_mirror_maker2.log:java.util.NoSuchElementException: None.get

        4. Also the kafka log sizes between source and target will not match:

        [/tmp] $ find kafka* -name *.kafka -ls
        19400444 6104 -rw-r--r-- 1 jfung eng 6246655 Jun 20 10:56 kafka-source4-logs/test01-0/00000000000000000000.kafka
        19400819 5356 -rw-r--r-- 1 jfung eng 5483627 Jun 20 10:56 kafka-target3-logs/test01-0/00000000000000000000.kafka

        5. Notes about the patch kafka-306-v2.patch: this patch fix the broker_failure test suite to do the followings:
        a. Start 4 kafka brokers as source cluster
        b. Start 3 kafka brokers as target cluster
        c. Start 3 mirror maker to enable mirroring
        d. Send n messages to source cluster
        e. No bouncing is performed in this test for simplicity
        f. After the producer is stopped, validate the data count is matched between source & target
        Exception in Mirror Maker log:
        =========================
        [2012-06-20 10:56:04,364] DEBUG Getting broker partition info for topic test01 (kafka.producer.BrokerPartitionInfo)
        [2012-06-20 10:56:04,365] INFO Fetching metadata for topic test01 (kafka.producer.BrokerPartitionInfo)
        [2012-06-20 10:56:04,366] ERROR Error in handling batch of 200 events (kafka.producer.async.ProducerSendThread)
        java.util.NoSuchElementException: None.get
                at scala.None$.get(Option.scala:185)
                at scala.None$.get(Option.scala:183)
                at kafka.producer.ProducerPool.getAnyProducer(ProducerPool.scala:76)
                at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:73)
                at kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:45)
                at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:129)
                at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:95)
                at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:94)
                at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
                at scala.collection.immutable.List.foreach(List.scala:45)
                at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:44)
                at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:42)
                at kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:94)
                at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:65)
                at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:49)
                at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:96)
                at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:82)
                at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:60)
                at scala.collection.immutable.Stream.foreach(Stream.scala:254)
                at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:59)
                at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:37)

        Steps to reproduce
        =================
        It cannot be reproduced consistently. However, running the following script 2 or 3 times (step 2) will show the error:

        1. Apply kafka-306-v2.patch to 0.8 branch (revision 1352192 is used to reproduce this)

        2. Under the directory <kafka home>/system_test/broker_failure, execute the following command:
        => $ bin/run-test.sh 5 0

        3. Check the log under the directory <kafka home>/system_test/broker_failure:
        => $ grep Exception `ls kafka_mirror_maker*.log`
        => kafka_mirror_maker2.log:java.util.NoSuchElementException: None.get

        4. Also the kafka log sizes between source and target will not match:

        [/tmp] $ find kafka* -name *.kafka -ls
        19400444 6104 -rw-r--r-- 1 jfung eng 6246655 Jun 20 10:56 kafka-source4-logs/test01-0/00000000000000000000.kafka
        19400819 5356 -rw-r--r-- 1 jfung eng 5483627 Jun 20 10:56 kafka-target3-logs/test01-0/00000000000000000000.kafka

        Notes about the patch kafka-306-v2.patch
        ===============================
        This patch fix the broker_failure test suite to do the followings:

        a. Start 4 kafka brokers as source cluster
        b. Start 3 kafka brokers as target cluster
        c. Start 3 mirror maker to enable mirroring
        d. Send n messages to source cluster
        e. No bouncing is performed in this test for simplicity
        f. After the producer is stopped, validate the data count is matched between source & target
        Hide
        Jun Rao added a comment -

        Attach patch v1. The issue is that the code relies on the broker ids in the hashmap to be always between 0 and size - 1, which is not always true, especially when there are failures.

        Show
        Jun Rao added a comment - Attach patch v1. The issue is that the code relies on the broker ids in the hashmap to be always between 0 and size - 1, which is not always true, especially when there are failures.
        Jun Rao made changes -
        Attachment kafka-370_v1.patch [ 12533071 ]
        Hide
        Joel Koshy added a comment -

        +1

        Show
        Joel Koshy added a comment - +1
        Hide
        Jun Rao added a comment -

        Thanks for the review. Committed to 0.8

        Show
        Jun Rao added a comment - Thanks for the review. Committed to 0.8
        Jun Rao made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee Jun Rao [ junrao ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Fixed [ 1 ]
        Hide
        John Fung added a comment -

        Downloaded rev. 1353005 and tried the test a few times. The issue is fixed.

        Show
        John Fung added a comment - Downloaded rev. 1353005 and tried the test a few times. The issue is fixed.
        John Fung made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1d 23h 3m 1 Jun Rao 22/Jun/12 18:32
        Resolved Resolved Closed Closed
        2h 43m 1 John Fung 22/Jun/12 21:16

          People

          • Assignee:
            Jun Rao
            Reporter:
            John Fung
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development