Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7879

Data directory size decreases every few minutes when producer is sending large amount of data

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I am running kafka broker with 6 nodes and have set reteion hours to 24 hours and retention bytes to 5 GB.

       

      I have set retention bytes to 250GB on  topic configuration.

       

      Now when producing message in async mode with 1000 bytes message with very high frequency. I am seeing that kafka data directory size increases but every 5 minutes it decreases by some percentage (in my observation it increases by 40G and then reduces to 20G, so in every 5 minutes we are seeing increase by 20G instead of 40G).

       

      Is there any extra configuration we need to set to avoid this data loss or is there some sort of compression that is going on here.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bansalp Pradeep Bansal
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: