Kafka
  1. Kafka
  2. KAFKA-1441

Purgatory purge causes latency spikes

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The request purgatory has a funky thing where it periodically loops over all watches and purges them. If you have a fair number of partitions you can accumulate lots of watches and purging them can take a long time. During this time all expiry is halted.

      Here is an example log:
      [2014-05-08 21:07:41,950] INFO ExpiredRequestReaper-2 Expired request after 10ms: 5829 (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:41,952] INFO ExpiredRequestReaper-2 Expired request after 10ms: 5882 (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:41,967] INFO ExpiredRequestReaper-2 Expired request after 11ms: 5884 (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:41,968] INFO ExpiredRequestReaper-2 Purging purgatory (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:41,969] INFO ExpiredRequestReaper-2 Purged 0 requests from delay queue. (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:42,305] INFO ExpiredRequestReaper-2 Purged 340809 (watcher) requests. (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:42,305] INFO ExpiredRequestReaper-2 Expired request after 106ms: 5847 (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:42,305] INFO ExpiredRequestReaper-2 Expired request after 106ms: 5904 (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:42,328] INFO ExpiredRequestReaper-2 Expired request after 10ms: 5908 (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:42,329] INFO ExpiredRequestReaper-2 Expired request after 10ms: 5852 (kafka.server.RequestPurgatory$ExpiredRequestReaper)
      [2014-05-08 21:07:42,343] INFO ExpiredRequestReaper-2 Expired request after 11ms: 5854 (kafka.server.RequestPurgatory$ExpiredRequestReaper)

      Combined with our buggy purgatory request impls that can sometimes hit their expiration this can lead to huge latency spikes.

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development