Kafka
  1. Kafka
  2. KAFKA-552

No error messages logged for those failing-to-send messages from Producer

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    1. testcase_0123_logs.tar.gz
      371 kB
      John Fung
    2. testcase_0122_logs.tar.gz
      376 kB
      John Fung
    3. system_test_output_1349730350.log
      224 kB
      John Fung

      Activity

      Hide
      John Fung added a comment -
      • Attached the following files for reference:

      File 1: system_test_output_1349730350.log - System Test console output file. It provides the sequence of all testing events with timestamps

      File 2: testcase_0122_logs.tar.gz - This file has all log messages for ZK, brokers, producer, consumer in testcase_0122. This testcase has producer in sync mode and acks set to 1

      File 3: testcase_0123_logs.tar.gz - This file has all log messages for ZK, brokers, producer, consumer in testcase_0123. This testcase has producer in async mode and acks set to 1

      • The issue in this JIRA is as follows (the following is taken from testcase_0123):

      1. In topic "test_2", the producer produces 6700 messages but consumer received 6695 messages (as shown below).
      2. In producer log, there are no Error messages related to the failure to send messages (producer log can be found in testcase_0123/logs/producer_performance-5/producer_performance.log in the attached tar file).
      3. From the log segment checksums shown below, messages are replicated to all brokers across the cluster properly.

      • Test Results shown for testcase_0123:

      2012-10-08 14:31:31,156 - INFO - ======================================================
      2012-10-08 14:31:31,156 - INFO - validating data matched
      2012-10-08 14:31:31,156 - INFO - ======================================================
      2012-10-08 14:31:31,241 - INFO - no. of unique messages on topic [test_1] sent from publisher : 6800 (kafka_system_test_utils)
      2012-10-08 14:31:31,241 - INFO - no. of unique messages on topic [test_1] received by consumer : 6800 (kafka_system_test_utils)
      2012-10-08 14:31:31,315 - INFO - no. of unique messages on topic [test_2] sent from publisher : 6700 (kafka_system_test_utils)
      2012-10-08 14:31:31,315 - INFO - no. of unique messages on topic [test_2] received by consumer : 6695 (kafka_system_test_utils)
      2012-10-08 14:31:31,315 - INFO - See /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_
      0123/logs/console_consumer-7/msg_id_missing_in_consumer.log for missing MessageID (kafka_system_test_utils)
      2012-10-08 14:31:31,316 - INFO - ================================================
      2012-10-08 14:31:31,317 - INFO - validating broker log segment checksums
      2012-10-08 14:31:31,317 - INFO - ================================================
      2012-10-08 14:31:38,996 - INFO - log segment files checksum :

      {u'broker-1':

      {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'}

      ,
      u'broker-2':

      {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'}

      ,
      u'broker-3': {'test_1-0/00000000000000000000.kafka': '691684159',
      'test_1-1/00000000000000000000.kafka': '3681205309',
      'test_1-2/00000000000000000000.kafka': '3571211770',
      'test_2-0/00000000000000000000.kafka': '85299972',
      'test_2-1/00000000000000000000.kafka': '4115697331',
      'test_2-2/00000000000000000000.kafka': '3974179431'}}

      Show
      John Fung added a comment - Attached the following files for reference: File 1: system_test_output_1349730350.log - System Test console output file. It provides the sequence of all testing events with timestamps File 2: testcase_0122_logs.tar.gz - This file has all log messages for ZK, brokers, producer, consumer in testcase_0122. This testcase has producer in sync mode and acks set to 1 File 3: testcase_0123_logs.tar.gz - This file has all log messages for ZK, brokers, producer, consumer in testcase_0123. This testcase has producer in async mode and acks set to 1 The issue in this JIRA is as follows (the following is taken from testcase_0123): 1. In topic "test_2", the producer produces 6700 messages but consumer received 6695 messages (as shown below). 2. In producer log, there are no Error messages related to the failure to send messages (producer log can be found in testcase_0123/logs/producer_performance-5/producer_performance.log in the attached tar file). 3. From the log segment checksums shown below, messages are replicated to all brokers across the cluster properly. Test Results shown for testcase_0123: 2012-10-08 14:31:31,156 - INFO - ====================================================== 2012-10-08 14:31:31,156 - INFO - validating data matched 2012-10-08 14:31:31,156 - INFO - ====================================================== 2012-10-08 14:31:31,241 - INFO - no. of unique messages on topic [test_1] sent from publisher : 6800 (kafka_system_test_utils) 2012-10-08 14:31:31,241 - INFO - no. of unique messages on topic [test_1] received by consumer : 6800 (kafka_system_test_utils) 2012-10-08 14:31:31,315 - INFO - no. of unique messages on topic [test_2] sent from publisher : 6700 (kafka_system_test_utils) 2012-10-08 14:31:31,315 - INFO - no. of unique messages on topic [test_2] received by consumer : 6695 (kafka_system_test_utils) 2012-10-08 14:31:31,315 - INFO - See /home/jfung/workspace_kafka/kafka_r1395693_502v7_sanity/system_test/replication_testsuite/testcase_ 0123/logs/console_consumer-7/msg_id_missing_in_consumer.log for missing MessageID (kafka_system_test_utils) 2012-10-08 14:31:31,316 - INFO - ================================================ 2012-10-08 14:31:31,317 - INFO - validating broker log segment checksums 2012-10-08 14:31:31,317 - INFO - ================================================ 2012-10-08 14:31:38,996 - INFO - log segment files checksum : {u'broker-1': {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'} , u'broker-2': {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'} , u'broker-3': {'test_1-0/00000000000000000000.kafka': '691684159', 'test_1-1/00000000000000000000.kafka': '3681205309', 'test_1-2/00000000000000000000.kafka': '3571211770', 'test_2-0/00000000000000000000.kafka': '85299972', 'test_2-1/00000000000000000000.kafka': '4115697331', 'test_2-2/00000000000000000000.kafka': '3974179431'}}

        People

        • Assignee:
          Unassigned
          Reporter:
          John Fung
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development