Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3979

Optimize memory used by replication process by using adaptive fetch message size

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: None
    • Component/s: replication
    • Labels:
      None
    • Flags:
      Patch

      Description

      Current replication process fetches messages in replica.fetch.max.bytes-sized chunks.

      Since replica.fetch.max.bytes should be bigger than max.message.bytes for replication to work, one can face big memory consumption for replication process, especially for installations with big number of partitions.

      Proposed solution is to try to fetch messages in smaller chunks (say replica.fetch.base.bytes).
      If we encounter message bigger than current fetch chunk, we increase chunk (f.e. twofold) and retry. After replicating this bigger message, we shrunk fetch chunk size back until it reaches replica.fetch.base.bytes

      replica.fetch.base.bytes should be chosen big enough not to affect throughput and to be bigger than most of messages.
      However, it can be much less than replica.fetch.max.bytes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                nepal Andrey Neporada
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: