[KAFKA-7235] Use brokerZkNodeVersion to prevent broker from processing outdated controller request

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Assignee: Zhanxiang (Patrick) Huang
• Reporter: Dong Lin
• Created: 02/Aug/18 06:31
• Updated: 30/Dec/19 22:21
• Resolved: 03/Dec/18 06:02

Description

Currently a broker can process controller requests that are sent before the broker is restarted. This could cause a few problems. Here is one example:

Let's assume partitions p1 and p2 exists on broker1.

1) Controller generates LeaderAndIsrRequest with p1 to be sent to broker1.

2) Before controller sends the request, broker1 is quickly restarted.

3) The LeaderAndIsrRequest with p1 is delivered to broker1.

4) After processing the first LeaderAndIsrRequest, broker1 starts to checkpoint high watermark for all partitions that it owns. Thus it may overwrite high watermark checkpoint file with only the hw for partition p1. The hw for partition p2 is now lost, which could be a problem.

In general, the correctness of broker logic currently relies on a few assumption, e.g. the first LeaderAndIsrRequest received by broker should contain all partitions hosted by the broker, which could break if broker can receive controller requests that were generated before it restarts. 

One reasonable solution to the problem is to include the expectedBrokeNodeZkVersion in the controller requests. Broker should remember the broker znode zkVersion after it registers itself in the zookeeper. Then broker can reject those controller requests whose expectedBrokeNodeZkVersion is different from its broker znode zkVersion.

 

Issue Links

• incorporates: KAFKA-1342 Slow controlled shutdowns can result in stale shutdown requests
• incorporates: KAFKA-3944 After the broker restart, fetchers stopped due to a delayed controlled shutdown message
• is duplicated by: KAFKA-1120 Controller could miss a broker state change
• links to: GitHub Pull Request #5821
• mentioned in: Page Loading...