Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-918

Change log.retention.hours to be log.retention.mins

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.2
    • Fix Version/s: 0.8.1
    • Component/s: config
    • Labels:

      Description

      We stood up a cluster that is processing over 350,000 events per second, with each event a fixed payload size of 2K. The storage required to process that much data over an hour is beyond what we wanted to pay for at AWS. Additionally, we don't have a requirement to keep the files around for an extended period after processing.

      It would be tremendously valuable for us to be able to define the log.retention in minutes, not hours. For example, we would prefer to only keep 30 minutes of logs around.

        Activity

        Hide
        jkreps Jay Kreps added a comment -

        This make sense, should be a pretty easy fix. We would definitely take a patch against trunk if could still support the older property as a fallback for compatibility.

        Show
        jkreps Jay Kreps added a comment - This make sense, should be a pretty easy fix. We would definitely take a patch against trunk if could still support the older property as a fallback for compatibility.
        Hide
        avasile Alin Vasile added a comment -

        Patch for trunk attached.

        Show
        avasile Alin Vasile added a comment - Patch for trunk attached.
        Hide
        junrao Jun Rao added a comment -

        Thanks for the patch. Looks good. Could you remove unused imports in KafkaConfigTest?

        Show
        junrao Jun Rao added a comment - Thanks for the patch. Looks good. Could you remove unused imports in KafkaConfigTest?
        Hide
        avasile Alin Vasile added a comment -

        New patch attached, unused imports are removed from KafkaConfigTest

        Show
        avasile Alin Vasile added a comment - New patch attached, unused imports are removed from KafkaConfigTest
        Hide
        junrao Jun Rao added a comment -

        +1 on the new patch. Committed to trunk.

        Show
        junrao Jun Rao added a comment - +1 on the new patch. Committed to trunk.
        Hide
        swapnilghike Swapnil Ghike added a comment - - edited

        We should get the property in one line instead of using a method, like Utils.parseCsvList(props.getString("log.dirs", props.getString("log.dir", "/tmp/kafka-logs"))).

        Show
        swapnilghike Swapnil Ghike added a comment - - edited We should get the property in one line instead of using a method, like Utils.parseCsvList(props.getString("log.dirs", props.getString("log.dir", "/tmp/kafka-logs"))).
        Hide
        avasile Alin Vasile added a comment -

        Wouldn't be better if both properties can be used in the same time? This won't break the previous functionality.

        Show
        avasile Alin Vasile added a comment - Wouldn't be better if both properties can be used in the same time? This won't break the previous functionality.

          People

          • Assignee:
            avasile Alin Vasile
            Reporter:
            jweiss Jason Weiss
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development