Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9952

Reconsider immediate propagation of hw when replica selector is used

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None

      Description

      For KIP-392, we added logic to make sure that high watermark changes are propagated to followers without delay in order to improve end to end latency when fetching from followers. The downside of this change is that it increases the rate of fetch requests from followers which can have a noticeable impact on performance (see KAFKA-9731). To fix that problem, we modified the code so that we only propagate high watermark changes immediately when a replica selector is used (which is not the default). However, we should consider whether we want to go even further and disable immediate propagation of the high watermark more generally since it makes this feature risky to use. Instead, users can use the max wait time in order to control the worst-case latency. Note that this is typically only a problem for low-throughput topics since otherwise we will have a steady rate of high watermark updates.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hachikuji Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: