Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-15906

Emit offset syncs more often than offset.lag.max for low-throughput/finite partitions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 3.7.0
    • mirrormaker
    • None

    Description

      Right now, the offset.lag.max configuration limits the number of offset syncs are emitted by the MirrorSourceTask, along with a fair rate-limiting semaphore. After 100 records have been emitted for a partition, and the semaphore is available, an offset sync can be emitted.

      For low-volume topics, the `offset.lag.max` default of 100 is much more restrictive than the rate-limiting semaphore. For example, a topic which mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset sync. If the topic is actually finite, the last offset sync will never arrive, and the translation will have a persistent lag.

      Instead, we can periodically flush the offset syncs for partitions that are under the offset.lag.max limit, but have not received an offset sync recently. This could be a new configuration, be a hard-coded time, or be based on the existing emit.checkpoints.interval.seconds and sync.group.offsets.interval.seconds configurations.

       

      Alternatively, we could decrease the default `offset.lag.max` value to 0, and rely on the fair semaphore to limit the number of syncs emitted for high-throughput partitions. The semaphore is not currently configurable, so users wanting lower throughput on the offset-syncs topic will still need an offset.lag.max > 0.

      Attachments

        Issue Links

          Activity

            People

              gharris1727 Greg Harris
              gharris1727 Greg Harris
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: