Kafka
  1. Kafka
  2. KAFKA-918

Change log.retention.hours to be log.retention.mins

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.2
    • Fix Version/s: 0.8.1
    • Component/s: config
    • Labels:

      Description

      We stood up a cluster that is processing over 350,000 events per second, with each event a fixed payload size of 2K. The storage required to process that much data over an hour is beyond what we wanted to pay for at AWS. Additionally, we don't have a requirement to keep the files around for an extended period after processing.

      It would be tremendously valuable for us to be able to define the log.retention in minutes, not hours. For example, we would prefer to only keep 30 minutes of logs around.

        Activity

        Hide
        Alin Vasile added a comment -

        Wouldn't be better if both properties can be used in the same time? This won't break the previous functionality.

        Show
        Alin Vasile added a comment - Wouldn't be better if both properties can be used in the same time? This won't break the previous functionality.
        Hide
        Swapnil Ghike added a comment - - edited

        We should get the property in one line instead of using a method, like Utils.parseCsvList(props.getString("log.dirs", props.getString("log.dir", "/tmp/kafka-logs"))).

        Show
        Swapnil Ghike added a comment - - edited We should get the property in one line instead of using a method, like Utils.parseCsvList(props.getString("log.dirs", props.getString("log.dir", "/tmp/kafka-logs"))).
        Hide
        Jun Rao added a comment -

        +1 on the new patch. Committed to trunk.

        Show
        Jun Rao added a comment - +1 on the new patch. Committed to trunk.
        Hide
        Alin Vasile added a comment -

        New patch attached, unused imports are removed from KafkaConfigTest

        Show
        Alin Vasile added a comment - New patch attached, unused imports are removed from KafkaConfigTest
        Hide
        Jun Rao added a comment -

        Thanks for the patch. Looks good. Could you remove unused imports in KafkaConfigTest?

        Show
        Jun Rao added a comment - Thanks for the patch. Looks good. Could you remove unused imports in KafkaConfigTest?
        Hide
        Alin Vasile added a comment -

        Patch for trunk attached.

        Show
        Alin Vasile added a comment - Patch for trunk attached.
        Hide
        Jay Kreps added a comment -

        This make sense, should be a pretty easy fix. We would definitely take a patch against trunk if could still support the older property as a fallback for compatibility.

        Show
        Jay Kreps added a comment - This make sense, should be a pretty easy fix. We would definitely take a patch against trunk if could still support the older property as a fallback for compatibility.

          People

          • Assignee:
            Alin Vasile
            Reporter:
            Jason Weiss
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development