Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3293

Consumers are not able to get messages.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 0.9.0.1
    • Fix Version/s: None
    • Component/s: consumer, offset manager
    • Labels:
      None
    • Environment:
      kafka: kafka_2.11-0.9.0.1
      java: jdk1.8.0_65
      OS: Linux stephen-T450s 3.19.0-51-generic #57~14.04.1-Ubuntu SMP Fri Feb 19 14:36:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

      Description

      Overview
      =======
      The results of test are not consistent.
      The problem is that something is preventing the consumer from receiving the messages.

      Configuration
      ==========
      Server (only num.partitions is changed)
      diff config/server.properties config.backup/server.properties
      65c65
      < num.partitions=8

      > num.partitions=1

      Producer
      properties.put("bootstrap.servers", “localhost:9092”);
      properties.put("acks", "all");
      properties.put("key.serializer", "org.apache.kafka.common.serialization.LongSerializer");
      properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
      properties.put("partitioner.class", "kafkatest.sample2.SimplePartitioner");

      Consumer
      properties.put("bootstrap.servers", “localhost:9092”);
      properties.put("group.id", "testGroup");
      properties.put("key.deserializer", "org.apache.kafka.common.serialization.LongDeserializer");
      properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
      properties.put("enable.auto.commit", "false");

      Steps to reproduce:
      ===============
      1. started the zookeeper
      2. started the kafka server
      3. created topic
      $ bin/kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partition 8 --topic testTopic4
      4. Ran SimpleProducerDriver with 5 producers, and the amount of messages produced is 50
      5. Offset Status
      $ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic testTopic4 --time -1
      testTopic4:2:1
      testTopic4:5:27
      testTopic4:4:1
      testTopic4:7:2
      testTopic4:1:8
      testTopic4:3:0
      testTopic4:6:11
      testTopic4:0:0
      6. waited till the producer driver completes, it takes no more than a few seconds
      7. ran the SimpleConsumerDriver a couple of times, and no message is received. Following DEBUG information is found:
      2016-02-25 22:42:19 DEBUG [pool-1-thread-2] Fetcher: - Ignoring fetched records for partition testTopic4-3 since it is no longer fetchable
      8. altered the properties of consumer, had the auto commit disabled:
      //properties.put("enable.auto.commit", "false");
      9. ran the SimpleConsumerDriver a couple of times, still, no message is received.
      Following DEBUG information is found:
      2016-02-25 22:47:23 DEBUG [pool-1-thread-2] ConsumerCoordinator: - Committed offset 8 for partition testTopic4-1
      seems like the offset was updated?
      10. re-enabled the auto commit, nothing changed.
      Following DEBUG information is found:
      2016-02-25 22:49:38 DEBUG [pool-1-thread-7] Fetcher: - Resetting offset for partition testTopic4-6 to the committed offset 11
      11. ran the SimpleProducerDriver again, another 50 messages are published
      12. ran the SimpleConsumerDriver again, 100 messages were consumed.
      13. ran the SimpleConsumerDriver again, 50 messages were consumed.
      As auto commit is disabled, all messages (100) should be consumed.

      The results of test are not consistent.
      The problem is that something is preventing the consumer from receiving the messages.
      And sometimes it required running the producer when the consumers are still active so as to get around it.
      And once the consumers started to consume messages, the problem did not occur any more.

        Attachments

          Activity

            People

            • Assignee:
              nehanarkhede Neha Narkhede
              Reporter:
              stephen.wong Stephen Wong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: