Kafka
  1. Kafka
  2. KAFKA-595

Decouple producer side compression from server-side compression.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      In 0.7 Kafka always appended messages to the log using whatever compression codec the client used. In 0.8, after the KAFKA-506 patch, the master always recompresses the message before appending to the log to assign ids. Currently the server uses a funky heuristic to choose a compression codec based on the codecs the producer used. This doesn't actually make that much sense. It would be better for the server to have its own compression (a global default and per-topic override) that specified the compression codec, and have the server always recompress with this codec regardless of the original codec.

      Compression currently happens in kafka.log.Log.assignOffsets (perhaps should be renamed if it takes on compression as an official responsibility instead of a side-effect).

        Activity

        Hide
        Neha Narkhede added a comment -

        >> I would recommend we instead add a log.compression.codec property (plus override map) that controls the compression on the broker.

        Yes, when I said move the compression to server side, that is what I meant. Also, I filed this JIRA to keep track of this optimization I came across while performance testing, don't think we should push this in 0.8

        Show
        Neha Narkhede added a comment - >> I would recommend we instead add a log.compression.codec property (plus override map) that controls the compression on the broker. Yes, when I said move the compression to server side, that is what I meant. Also, I filed this JIRA to keep track of this optimization I came across while performance testing, don't think we should push this in 0.8

          People

          • Assignee:
            Unassigned
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development