Kafka
  1. Kafka
  2. KAFKA-572

Merged log segment checksums mismatched in Leader failure System Test case

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Activity

      Hide
      John Fung added a comment -
      • Test Description:

      1. Start a 3-broker cluster as source
      2. Send messages to source cluster
      3. Find leader and terminate it (kill -15)
      4. Start the broker shortly
      5. Start a consumer to consume data
      6. Compare the MessageID in the data between producer log and consumer log.

      • To reproduce this issue, please do the followings:

      1. Download the latest 0.8 branch
      2. Apply the patch attached to this JIRA
      3. Build kafka by running "./sbt update package"
      4. Execute the test in directory "system_test" : "python -B system_test_runner.py"

      • Output from the test - No data loss but merged log segment checksums mismatched

      2012-10-12 22:27:35,760 - INFO - ======================================================
      2012-10-12 22:27:35,760 - INFO - validating data matched
      2012-10-12 22:27:35,760 - INFO - ======================================================
      2012-10-12 22:27:35,771 - INFO - no. of unique messages on topic [test_1] sent from publisher : 500 (kafka_system_test_utils)
      2012-10-12 22:27:35,771 - INFO - no. of unique messages on topic [test_1] received by consumer : 500 (kafka_system_test_utils)
      2012-10-12 22:27:35,771 - INFO - ================================================
      2012-10-12 22:27:35,771 - INFO - validating merged broker log segment checksums
      2012-10-12 22:27:35,771 - INFO - ================================================

      {u'kafka_server_1_logs:test_1-0': 'd70c8d37634b0b08cd407eb042e77ef8', u'kafka_server_2_logs:test_1-0': 'd70c8d37634b0b08cd407eb042e77ef8', u'kafka_server_3_logs:test_1-0': '924d30d5f0d2a8ba9ef45f7cba88e192'}

      2012-10-12 22:27:35,774 - ERROR - merged log segment checksum in test_1-0 mismatched (kafka_system_test_utils)

      Show
      John Fung added a comment - Test Description: 1. Start a 3-broker cluster as source 2. Send messages to source cluster 3. Find leader and terminate it (kill -15) 4. Start the broker shortly 5. Start a consumer to consume data 6. Compare the MessageID in the data between producer log and consumer log. To reproduce this issue, please do the followings: 1. Download the latest 0.8 branch 2. Apply the patch attached to this JIRA 3. Build kafka by running "./sbt update package" 4. Execute the test in directory "system_test" : "python -B system_test_runner.py" Output from the test - No data loss but merged log segment checksums mismatched 2012-10-12 22:27:35,760 - INFO - ====================================================== 2012-10-12 22:27:35,760 - INFO - validating data matched 2012-10-12 22:27:35,760 - INFO - ====================================================== 2012-10-12 22:27:35,771 - INFO - no. of unique messages on topic [test_1] sent from publisher : 500 (kafka_system_test_utils) 2012-10-12 22:27:35,771 - INFO - no. of unique messages on topic [test_1] received by consumer : 500 (kafka_system_test_utils) 2012-10-12 22:27:35,771 - INFO - ================================================ 2012-10-12 22:27:35,771 - INFO - validating merged broker log segment checksums 2012-10-12 22:27:35,771 - INFO - ================================================ {u'kafka_server_1_logs:test_1-0': 'd70c8d37634b0b08cd407eb042e77ef8', u'kafka_server_2_logs:test_1-0': 'd70c8d37634b0b08cd407eb042e77ef8', u'kafka_server_3_logs:test_1-0': '924d30d5f0d2a8ba9ef45f7cba88e192'} 2012-10-12 22:27:35,774 - ERROR - merged log segment checksum in test_1-0 mismatched (kafka_system_test_utils)
      Hide
      John Fung added a comment -
      • Data Log Segment files sizes:

      system_test/replication_testsuite/testcase_0102/logs $ find broker-* -name '00*.log' -ls

      9702225 12 rw-rr- 1 jfung eng 10279 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000301.log
      9702226 12 rw-rr- 1 jfung eng 10271 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000000.log
      9702227 12 rw-rr- 1 jfung eng 10293 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000201.log
      9702228 12 rw-rr- 1 jfung eng 10292 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000101.log
      9702230 12 rw-rr- 1 jfung eng 10178 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000401.log

      9702239 12 rw-rr- 1 jfung eng 10279 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000301.log
      9702240 12 rw-rr- 1 jfung eng 10271 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000000.log
      9702241 12 rw-rr- 1 jfung eng 10293 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000201.log
      9702242 12 rw-rr- 1 jfung eng 10292 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000101.log
      9702244 12 rw-rr- 1 jfung eng 10178 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000401.log

      9702252 12 rw-rr- 1 jfung eng 10279 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000361.log
      9702255 4 rw-rr- 1 jfung eng 4010 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000461.log
      9702256 12 rw-rr- 1 jfung eng 10271 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000000.log
      9702257 8 rw-rr- 1 jfung eng 5657 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000101.log
      9702259 12 rw-rr- 1 jfung eng 10280 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000261.log
      9702262 12 rw-rr- 1 jfung eng 10816 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000156.log

      Show
      John Fung added a comment - Data Log Segment files sizes: system_test/replication_testsuite/testcase_0102/logs $ find broker-* -name '00*.log' -ls 9702225 12 rw-r r - 1 jfung eng 10279 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000301.log 9702226 12 rw-r r - 1 jfung eng 10271 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000000.log 9702227 12 rw-r r - 1 jfung eng 10293 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000201.log 9702228 12 rw-r r - 1 jfung eng 10292 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000101.log 9702230 12 rw-r r - 1 jfung eng 10178 Oct 12 22:27 broker-1/kafka_server_1_logs/test_1-0/00000000000000000401.log 9702239 12 rw-r r - 1 jfung eng 10279 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000301.log 9702240 12 rw-r r - 1 jfung eng 10271 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000000.log 9702241 12 rw-r r - 1 jfung eng 10293 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000201.log 9702242 12 rw-r r - 1 jfung eng 10292 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000101.log 9702244 12 rw-r r - 1 jfung eng 10178 Oct 12 22:27 broker-2/kafka_server_2_logs/test_1-0/00000000000000000401.log 9702252 12 rw-r r - 1 jfung eng 10279 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000361.log 9702255 4 rw-r r - 1 jfung eng 4010 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000461.log 9702256 12 rw-r r - 1 jfung eng 10271 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000000.log 9702257 8 rw-r r - 1 jfung eng 5657 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000101.log 9702259 12 rw-r r - 1 jfung eng 10280 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000261.log 9702262 12 rw-r r - 1 jfung eng 10816 Oct 12 22:27 broker-3/kafka_server_3_logs/test_1-0/00000000000000000156.log
      Hide
      John Fung added a comment -

      This issue is due to a bug in the System Test script (log segment files were not sorted before merging). So mark this Fixed.

      Show
      John Fung added a comment - This issue is due to a bug in the System Test script (log segment files were not sorted before merging). So mark this Fixed.

        People

        • Assignee:
          Unassigned
          Reporter:
          John Fung
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development