[KAFKA-1430] Purgatory redesign

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.8.2.0
• Fix Version/s: 0.8.2.0
• Component/s: core
• Assignee: Guozhang Wang
• Reporter: Jun Rao
• Created: 30/Apr/14 00:23
• Updated: 30/Jul/19 08:03
• Resolved: 07/Oct/14 00:28

Description

We have seen 2 main issues with the Purgatory.

1. There is no atomic checkAndWatch functionality. So, a client typically first checks whether a request is satisfied or not and then register the watcher. However, by the time the watcher is registered, the registered item could already be satisfied. This item won't be satisfied until the next update happens or the delayed time expires, which means the watched item could be delayed.

2. FetchRequestPurgatory doesn't quite work. This is because the current design tries to incrementally maintain the accumulated bytes ready for fetch. However, this is difficult since the right time to check whether a fetch (for regular consumer) request is satisfied is when the high watermark moves. At that point, it's hard to figure out how many bytes we should incrementally add to each pending fetch request.

The problem has been reported in KAFKA-1150 and KAFKA-703.

Issue Links

• contains: KAFKA-703 A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request
• contains: KAFKA-1150 Fetch on a replicated topic does not return as soon as possible
• contains: KAFKA-1441 Purgatory purge causes latency spikes
• is duplicated by: KAFKA-1150 Fetch on a replicated topic does not return as soon as possible
• supercedes: KAFKA-1016 Broker should limit purgatory size