Kafka
  1. Kafka
  2. KAFKA-164

Config should default to a higher-throughput configuration for log.flush.interval

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: config
    • Labels:
      None

      Description

      Currently we default the flush interval to log.flush.interval=1. This is very slow as it immediately flushes each message. I recommend we change this to 20000 and drop the time-based flush to 1 second. This should be a good default trade-off between latency and throughput.

      1. KAFKA-164-v2.patch
        10 kB
        Jay Kreps
      2. KAFKA-164.patch
        0.7 kB
        Jay Kreps

        Activity

        Hide
        Jun Rao added a comment -

        Maybe shrink flush time to 500ms?

        Show
        Jun Rao added a comment - Maybe shrink flush time to 500ms?
        Hide
        Chris Burroughs added a comment -

        There are good things to be said about having the default configuration be the "overly paranoid do not loose your data" one, even though once they think about what that means no one actually wants that. I'd stick with round numbers and a note elaborating on the trade-offs.

        Show
        Chris Burroughs added a comment - There are good things to be said about having the default configuration be the "overly paranoid do not loose your data" one, even though once they think about what that means no one actually wants that. I'd stick with round numbers and a note elaborating on the trade-offs.
        Hide
        Jay Kreps added a comment -

        I added documentation on the tradeoffs for flush policy. I also went back and improved the documention across the board for the options we have and added in commented out stubs for options that were missing but are available (such as size-based retention policy).

        I find no one ever reads the code to find out about configs so the config file is a pretty good place to document them.

        Show
        Jay Kreps added a comment - I added documentation on the tradeoffs for flush policy. I also went back and improved the documention across the board for the options we have and added in commented out stubs for options that were missing but are available (such as size-based retention policy). I find no one ever reads the code to find out about configs so the config file is a pretty good place to document them.
        Hide
        Jay Kreps added a comment -

        Err, ignore the random Jmx file in the patch...left over from another commit.

        Show
        Jay Kreps added a comment - Err, ignore the random Jmx file in the patch...left over from another commit.
        Hide
        Jun Rao added a comment -

        +1 on changes in server.properties

        Show
        Jun Rao added a comment - +1 on changes in server.properties

          People

          • Assignee:
            Jay Kreps
            Reporter:
            Jay Kreps
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development