Kafka
  1. Kafka
  2. KAFKA-716

SimpleConsumerPerformance does not consume all available messages

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      To reproduce the issue:

      1. Start 1 zookeeper

      2. Start 1 broker

      3. Send some messages

      4. Start SimpleConsumerPerformance to consume messages. The only way to consume all messages is to set the fetch-size to be greater than the log segment file size.

      5. This output shows that SimpleConsumerPerformance consumes only 6 messages:

      $ bin/kafka-run-class.sh kafka.perf.SimpleConsumerPerformance --server kafka://host1:9092 --topic topic_001 --fetch-size 2048 --partition 0
      start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
      2013-01-21 15:09:21:124, 2013-01-21 15:09:21:165, 2048, 0.0059, 0.1429, 6, 146.3415

      6. This output shows that ConsoleConsumer consumes all 5500 messages (same test as the above)

      $ bin/kafka-run-class.sh kafka.consumer.ConsoleConsumer --zookeeper host2:2181 --topic topic_001 --consumer-timeout-ms 5000 --formatter kafka.consumer.ChecksumMessageFormatter --from-beginning | grep ^checksum | wc -l
      Consumed 5500 messages
      5500

        Activity

        John Fung made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        John Fung made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        John Fung added a comment -

        Not an issue

        Show
        John Fung added a comment - Not an issue
        Neha Narkhede made changes -
        Labels p2
        Neha Narkhede made changes -
        Field Original Value New Value
        Priority Major [ 3 ] Blocker [ 1 ]
        John Fung created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            John Fung
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development