[KAFKA-9089] Reassignment should be resilient to unexpected errors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4.0
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Created: 23/Oct/19 17:41
• Updated: 25/Oct/19 02:35
• Resolved: 25/Oct/19 02:35

Description

Reassignment changes typically involve both an update to the assignment state in zookeeper and an update to the in-memory representation of that state (in the ControllerContext). We can run into trouble when these states get inconsistent with each other, so the reassignment logic attempts to follow some rules to reduce the impact from this:

• When creating a new reassignment, we update the state in zookeeper first before updating memory. Until the reassignment is known to be persisted, we do not begin executing any reassignment logic.
• When completing a reassignment, all of the completion steps are executed before the state is updated in zookeeper. In the event of a failure, the new controller can retry reassignment completion.

However, the current logic does not follow these rules strictly which can lead to state inconsistencies in the case of an unexpected error.

1. When we override or cancel an existing assignment, we currently use an intermediate assignment state which is only reflected in memory. It is basically a mix of the previous assignment state and the overlapping parts of the new reassignment. The purpose of this is to shutdown unneeded replicas from the existing reassignment. Since the intermediate state is not persisted, a controller failure will revert to the old reassignment. Any exception which does not cause a controller failure will result in state divergence.
2. The target replicas of a reassignment are represented both in the existing assignment (PartitionReplicaAssignment) and in a separate context object (ReassignedPartitionContext). The reassignment context is updated before a reassignment has been accepted and persisted. The intent is to remove this context object in the event of a submission failure, but an unexpected error will leave it around.

We can make reassignment more resilient to unexpected errors by using consistent update invariants. Specifically we can remove the intermediate assignment state and enforce the invariant that any active reassignment must be persisted before being reflected in memory. Additionally, we can make the assignment state the source of truth for the target replicas and eliminate the possibility of inconsistency. Doing so simplifies the reassignment logic and makes it more resilient.

Issue Links

• links to: GitHub Pull Request #7562