[KAFKA-5307] Concurrent offset commit on same partition may result in inconsistent cache

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: offset manager
• Assignee: Unassigned
• Reporter: Jason Gustafson
• Created: 22/May/17 21:18
• Updated: 03/Aug/18 00:10

Description

Consider two concurrent offset commits: the first one at offset A, the second at offset B. It is possible for both offsets to be appended to the log before either of them had been successfully acknowledged. Currently we maintain in the group metadata a collection of pending offset commits, but we only store one offset for each partition. When the commit for A is received, we add an entry for that partition with A. When the commit for B arrives, we would overwrite that value with B. Now two cases where this results in incorrect behavior.

1. After the callback for A is invoked, we materialize whatever offset is in the pending offset collection unconditionally. In this case, we would incorrectly materialize B even though it had not been successfully written. Later B may fail and we have inconsistent state.
2. It could happen that the callback for B is invoked before A. If the result was successful, no harm done, but if it failed, then we need to restore A since it may still be completed successfully. If it does complete successfully, then we'll have inconsistent state again.