Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8808

Kafka Inconsistent Retention Period across partitions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: log, log cleaner
    • Labels:
      None

      Description

      Our topic is created with  retention period of 3 days.  Topic has four partitions.  Broker level default is 12 hour. 

      Some partition's segment get deleted even before 3 days. Logs show that segment is marked for deletion because it exceeded 43200000ms  = 12 hours. 

       

      "INFO Found deletable segments with base offsets [43275] due to retention time 43200000ms breach (kafka.log.Log)"

       

      This does not happen for all partitions.  Post full cluster bounce , partitions facing this issue change. 

       

      Topic config : 

      Topic:TOPICNAME PartitionCount:4 ReplicationFactor:2 Configs:retention.ms=259200000,segment.ms=43200000
      Topic: TOPICNAME Partition: 0 Leader: 1 Replicas: 1,8 Isr: 1,8
      Topic: TOPICNAME Partition: 1 Leader: 2 Replicas: 2,9 Isr: 9,2
      Topic: TOPICNAME Partition: 2 Leader: 3 Replicas: 3,1 Isr: 1,3
      Topic: TOPICNAME Partition: 3 Leader: 4 Replicas: 4,2 Isr: 4,2

       

      Broker config :

      log.retention.ms=43200000

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              psawant Prashant
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: