Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10268

dynamic config like "--delete-config log.retention.ms" doesn't work

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.6.0, 2.7.0
    • Component/s: log, log cleaner
    • Labels:
      None

      Description

      After I set "log.retention.ms=301000" to clean the data,i use the cmd

      "bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --delete-config log.retention.ms" to reset to default.

      Static broker configuration like log.retention.hours is 168h and no topic level configuration like retention.ms.

      it did not take effect actually although server.log print the broker configuration like that.

      log.retention.check.interval.ms = 300000
      log.retention.hours = 168
      log.retention.minutes = null
      log.retention.ms = null
      log.roll.hours = 168
      log.roll.jitter.hours = 0
      log.roll.jitter.ms = null
      log.roll.ms = null
      log.segment.bytes = 1073741824
      log.segment.delete.delay.ms = 60000

       

      Then we can see that retention time is still 301000ms from the server.log and segments have been deleted.

      [2020-07-13 14:30:00,958] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Found deletable segments with base offsets [5005329,6040360] due to retention time 301000ms breach (kafka.log.Log)
      [2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005329, size 1073741222] for deletion. (kafka.log.Log)
      [2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040360, size 1073728116] for deletion. (kafka.log.Log)
      [2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Incrementing log start offset to 7075648 (kafka.log.Log)
      [2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Found deletable segments with base offsets [5005330,6040410] due to retention time 301000ms breach (kafka.log.Log)
      [2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005330, size 1073732368] for deletion. (kafka.log.Log)
      [2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040410, size 1073735366] for deletion. (kafka.log.Log)
      [2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Incrementing log start offset to 7075685 (kafka.log.Log)
      [2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Deleting segment 5005329 (kafka.log.Log)
      [2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Deleting segment 6040360 (kafka.log.Log)
      [2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Deleting segment 5005330 (kafka.log.Log)
      [2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Deleting segment 6040410 (kafka.log.Log)
      [2020-07-13 14:31:01,144] INFO Deleted log /data/kafka_logs-test/test_retention-2/00000000000006040360.log.deleted. (kafka.log.LogSegment)
      [2020-07-13 14:31:01,144] INFO Deleted offset index /data/kafka_logs-test/test_retention-2/00000000000006040360.index.deleted. (kafka.log.LogSegment)
      [2020-07-13 14:31:01,144] INFO Deleted time index /data/kafka_logs-test/test_retention-2/00000000000006040360.timeindex.deleted. (kafka.log.LogSegment)

       

      Here are a few steps to reproduce it.

      1、set log.retention.ms=301000:

      bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --add-config log.retention.ms=301000

      2、produce messages to the topic:

      bin/kafka-producer-perf-test.sh --topic test_retention --num-records 10000000 --throughput -1 --producer-props bootstrap.servers=10.129.104.15:9092 --record-size 1024

      3、reset log.retention.ms to the default:

      bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --delete-config log.retention.ms

       

      I have attched server.log. You can see the log from row 238 to row 731. 

        Attachments

        1. server.log.2020-07-13-14
          104 kB
          zhifeng.peng

          Issue Links

            Activity

              People

              • Assignee:
                huxi_2b huxihx
                Reporter:
                zhifeng.peng zhifeng.peng
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: