Kafka
  1. Kafka
  2. KAFKA-662

Create testcases for unclean shut down

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Activity

      Hide
      John Fung added a comment -

      2. testcase_9072 - This testcase is to test the basic behavior of unclean shutdown where log truncation takes place.

      B1 B2
      ======= =======

      a. Both brokers are up running

      b. Send 5 messages

      m0 m0
      m1 m1
      m2 m2
      m3 m3
      m4 m4

      c. Shut down B2

      d. Send 10 messages

      m5
      m6
      m7
      m8
      m9
      m10
      m11
      m12
      m13
      m14

      e. Shut down B1 (B1 & B2 are both down)

      f. Sleep 5 seconds

      g. Start B2 (B1 is still down)

      h. Send 5 messages

      m15
      m16
      m17
      m18
      m19

      i. Start B1 (both B1 & B2 are up running)

      Recovering ...

      j. Since there are discrepancy in no. of messages maintained by the internal "index", there will be log truncation in B1's data log:

      m0 m0
      m1 m1
      m2 m2
      m3 m3
      m4 m4
      m15 m15
      m16 m16
      m17 m17
      m18 m18
      m19 m19

      $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log
      Dumping /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log
      Starting offset: 0
      offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000
      offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001
      offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002
      offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003
      offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004
      offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1414561216 payload: Topic:t001:ThreadID:0:MessageID:0000000015
      offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4018435027 payload: Topic:t001:ThreadID:0:MessageID:0000000016
      offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2261284386 payload: Topic:t001:ThreadID:0:MessageID:0000000017
      offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3219081918 payload: Topic:t001:ThreadID:0:MessageID:0000000018
      offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3599978319 payload: Topic:t001:ThreadID:0:MessageID:0000000019

      $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log
      Dumping /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log
      Starting offset: 0
      offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000
      offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001
      offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002
      offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003
      offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004
      offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1414561216 payload: Topic:t001:ThreadID:0:MessageID:0000000015
      offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4018435027 payload: Topic:t001:ThreadID:0:MessageID:0000000016
      offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2261284386 payload: Topic:t001:ThreadID:0:MessageID:0000000017
      offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3219081918 payload: Topic:t001:ThreadID:0:MessageID:0000000018
      offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3599978319 payload: Topic:t001:ThreadID:0:MessageID:0000000019

      Show
      John Fung added a comment - 2. testcase_9072 - This testcase is to test the basic behavior of unclean shutdown where log truncation takes place. B1 B2 ======= ======= a. Both brokers are up running b. Send 5 messages m0 m0 m1 m1 m2 m2 m3 m3 m4 m4 c. Shut down B2 d. Send 10 messages m5 m6 m7 m8 m9 m10 m11 m12 m13 m14 e. Shut down B1 (B1 & B2 are both down) f. Sleep 5 seconds g. Start B2 (B1 is still down) h. Send 5 messages m15 m16 m17 m18 m19 i. Start B1 (both B1 & B2 are up running) Recovering ... j. Since there are discrepancy in no. of messages maintained by the internal "index", there will be log truncation in B1's data log: m0 m0 m1 m1 m2 m2 m3 m3 m4 m4 m15 m15 m16 m16 m17 m17 m18 m18 m19 m19 $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log Dumping /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log Starting offset: 0 offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000 offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001 offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002 offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003 offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004 offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1414561216 payload: Topic:t001:ThreadID:0:MessageID:0000000015 offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4018435027 payload: Topic:t001:ThreadID:0:MessageID:0000000016 offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2261284386 payload: Topic:t001:ThreadID:0:MessageID:0000000017 offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3219081918 payload: Topic:t001:ThreadID:0:MessageID:0000000018 offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3599978319 payload: Topic:t001:ThreadID:0:MessageID:0000000019 $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log Dumping /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log Starting offset: 0 offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000 offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001 offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002 offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003 offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004 offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1414561216 payload: Topic:t001:ThreadID:0:MessageID:0000000015 offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4018435027 payload: Topic:t001:ThreadID:0:MessageID:0000000016 offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2261284386 payload: Topic:t001:ThreadID:0:MessageID:0000000017 offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3219081918 payload: Topic:t001:ThreadID:0:MessageID:0000000018 offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3599978319 payload: Topic:t001:ThreadID:0:MessageID:0000000019
      Hide
      John Fung added a comment -

      1. testcase_9071 - This testcase is to test the basic behavior of unclean shutdown where no log truncation takes place.

      B1 B2
      ======= =======

      a. Both brokers are up running

      b. Send 5 messages

      m0 m0
      m1 m1
      m2 m2
      m3 m3
      m4 m4

      c. Shut down B2

      d. Send 5 messages

      m5
      m6
      m7
      m8
      m9

      e. Shut down B1 (B1 & B2 are both down)

      f. Sleep 5 seconds

      g. Start B2 (B1 is still down)

      h. Send 5 messages

      m10
      m11
      m12
      m13
      m14

      i. Start B1 (both B1 & B2 are up running)

      Recovering ...

      j. Since there are no discrepancy in no. of messages maintained by the internal "index", there will not be any log truncation:

      m0 m0
      m1 m1
      m2 m2
      m3 m3
      m4 m4
      m5 m10
      m6 m11
      m7 m12
      m8 m13
      m9 m14

      $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log
      Dumping /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log
      Starting offset: 0
      offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000
      offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001
      offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002
      offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003
      offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004
      offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2659447631 payload: Topic:t001:ThreadID:0:MessageID:0000000005
      offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 626523484 payload: Topic:t001:ThreadID:0:MessageID:0000000006
      offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1276901549 payload: Topic:t001:ThreadID:0:MessageID:0000000007
      offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1963770929 payload: Topic:t001:ThreadID:0:MessageID:0000000008
      offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 474005952 payload: Topic:t001:ThreadID:0:MessageID:0000000009

      $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log
      Dumping /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log
      Starting offset: 0
      offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000
      offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001
      offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002
      offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003
      offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004
      offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1130195380 payload: Topic:t001:ThreadID:0:MessageID:0000000010
      offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 705784901 payload: Topic:t001:ThreadID:0:MessageID:0000000011
      offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2445632086 payload: Topic:t001:ThreadID:0:MessageID:0000000012
      offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4169754535 payload: Topic:t001:ThreadID:0:MessageID:0000000013
      offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1025275953 payload: Topic:t001:ThreadID:0:MessageID:0000000014

      Show
      John Fung added a comment - 1. testcase_9071 - This testcase is to test the basic behavior of unclean shutdown where no log truncation takes place. B1 B2 ======= ======= a. Both brokers are up running b. Send 5 messages m0 m0 m1 m1 m2 m2 m3 m3 m4 m4 c. Shut down B2 d. Send 5 messages m5 m6 m7 m8 m9 e. Shut down B1 (B1 & B2 are both down) f. Sleep 5 seconds g. Start B2 (B1 is still down) h. Send 5 messages m10 m11 m12 m13 m14 i. Start B1 (both B1 & B2 are up running) Recovering ... j. Since there are no discrepancy in no. of messages maintained by the internal "index", there will not be any log truncation: m0 m0 m1 m1 m2 m2 m3 m3 m4 m4 m5 m10 m6 m11 m7 m12 m8 m13 m9 m14 $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log Dumping /tmp/kafka_server_1_logs/t001-0/00000000000000000000.log Starting offset: 0 offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000 offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001 offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002 offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003 offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004 offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2659447631 payload: Topic:t001:ThreadID:0:MessageID:0000000005 offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 626523484 payload: Topic:t001:ThreadID:0:MessageID:0000000006 offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1276901549 payload: Topic:t001:ThreadID:0:MessageID:0000000007 offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1963770929 payload: Topic:t001:ThreadID:0:MessageID:0000000008 offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 474005952 payload: Topic:t001:ThreadID:0:MessageID:0000000009 $ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log Dumping /tmp/kafka_server_2_logs/t001-0/00000000000000000000.log Starting offset: 0 offset: 0 position: 0 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2307843899 payload: Topic:t001:ThreadID:0:MessageID:0000000000 offset: 1 position: 126 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 3770868426 payload: Topic:t001:ThreadID:0:MessageID:0000000001 offset: 2 position: 252 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1528229081 payload: Topic:t001:ThreadID:0:MessageID:0000000002 offset: 3 position: 378 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 844818728 payload: Topic:t001:ThreadID:0:MessageID:0000000003 offset: 4 position: 504 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4157601470 payload: Topic:t001:ThreadID:0:MessageID:0000000004 offset: 5 position: 630 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1130195380 payload: Topic:t001:ThreadID:0:MessageID:0000000010 offset: 6 position: 756 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 705784901 payload: Topic:t001:ThreadID:0:MessageID:0000000011 offset: 7 position: 882 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 2445632086 payload: Topic:t001:ThreadID:0:MessageID:0000000012 offset: 8 position: 1008 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 4169754535 payload: Topic:t001:ThreadID:0:MessageID:0000000013 offset: 9 position: 1134 isvalid: true payloadsize: 100 magic: 0 compresscodec: NoCompressionCodec crc: 1025275953 payload: Topic:t001:ThreadID:0:MessageID:0000000014

        People

        • Assignee:
          John Fung
          Reporter:
          John Fung
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:

            Development