Details
-
Improvement
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
3.1.0
-
None
-
None
Description
For fetch request version 13 we have the current method of handling unknown topic IDs.
- When the receiving broker sees an unknown topic ID in a request or encounters an inconsistent (mismatched) ID in the logs, it sends a top-level error back, delays all partitions (in fetcher thread), and closes the session
Ideally, we want to handle the same way as unknown topic names. We hold the topic partition in the session and try to resolve on a future fetch request.
However, there are a few complications with this approach and this is why we opted to simply close the session. One is that many objects in the fetch session are keyed using topic name+partition. We also still need the topic name for a few other checks in the fetch path.
Still, upon further inspection, closing the session on any new topics (when using topic names, we often see a few unknown topic or partition exceptions) is not ideal.
One way to see similar behavior in the topic ID case is to store unresolved IDs in the session. For compatibility reasons, we will need to add and/or generify a few classes. The general idea is that for sessions using version 13+ we will use a structure that keys using topic ID and partition with an optional name. That way, we can send partition level errors for unknown topic IDs and handle them later in the same session. We can also remove partitions using topic ID more easily if the unknown topic ID was due to stale metadata.
Then the new method will be as follows:
- When the receiving broker sees an unknown topic ID in a request or encounters an inconsistent (mismatched) ID in the logs, it sends an error on the unknown partition, delay only those partitions (in fetcher thread), and keep the session open to try to resolve in the future
Attachments
Issue Links
- links to