[KAFKA-4443] Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1.0
• Fix Version/s: 0.10.2.0
• Labels: reliability
• Assignee: Dong Lin
• Reporter: Dong Lin
• Created: 25/Nov/16 04:20
• Updated: 01/Dec/16 15:41
• Resolved: 29/Nov/16 22:34

Description

Currently in onControllerFailover(), controller will startup replicaStatemachine and partitionStateMachine before invoking sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq). However, if a broker starts right after controller election, the LeaderAndIsrRequest sent to follower partitions on this broker will all be ignored because broker doesn't know the leaders are alive.

To fix this problem, in onControllerFailover(), controller should send UpdateMetadataRequest to brokers after initializeControllerContext() but before it starts replicaStatemachine and partitionStateMachine. The first MetadatUpdateRequest will include list of live broker. Although it will not include partition leader information, it is OK because we will always send MetadataUpdateRequest again when we send LeaderAndIsrRequest during replicaStateMachine.startup() and partitionStateMachine.startup().

Issue Links

• links to: GitHub Pull Request #2168
• links to: GitHub Pull Request #2194