[FLINK-26522] FLIP-285: Refactoring code for multiple component leader election

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.18.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Matthias Pohl
• Reporter: Niklas Semmler
• Created: 07/Mar/22 15:04
• Updated: 30/Aug/23 07:54
• Resolved: 10/Jul/23 12:45

Description

The current implementation of the multiple component leader election faces a number of issues. These issues mostly stem from an attempt to make the multiple leader election process work just the same way as the single component leader election.

An attempt at listing the issues follows:

• Naming MultipleComponentLeaderElectionService appears by name similar to the LeaderElectionService, but is in fact closer to the LeaderElectionDriver.
• Similarity The interfaces LeaderElectionService, LeaderElectionDriver and MultipleComponentLeaderElectionDriver are very similar to each other.
• Cyclic dependency DefaultMultipleComponentLeaderElectionService holds a reference to the ZooKeeperMultipleComponentLeaderElectionDriver (MultipleComponentLeaderElectionDriver), which in turn holds a reference to the DefaultMultipleComponentLeaderElectionService (LeaderLatchListener)
• Unclear contract With single component leader election drivers such as ZooKeeperLeaderElectionDriver a call to the LeaderElectionService#stop from JobMasterServiceLeadershipRunner#closeAsync implies giving up the leadership of the JobMaster. With the multiple component leader election this is no longer the case. The leadership is held until the HighAvailabilityServices shutdown. This logic may be difficult to understand from the perspective of one of the components (e.g., the Dispatcher)
• Long call hierarchy DefaultLeaderElectionService->MultipleComponentLeaderElectionDriverAdapter->MultipleComponentLeaderElectionService->ZooKeeperMultipleComponentLeaderElectionDriver
• Long prefix "MultipleComponentLeaderElection" is quite a long prefix but shared by many classes.
• Adapter as primary implementation All non-testing non-multiple-component leadership drivers are deprecated. The primary implementation of LeaderElectionDriver is the adapter MultipleComponentLeaderElectionDriverAdapter.
• Possible redundancy We currently have similar methods for the Dispatcher, ResourceManager, JobMaster and WebMonitorEndpoint. (E.g., for granting leadership.) As these methods are called at the same time due to the multiple component leader election, it may make sense to combine this logic into a single object.

Issue Links

• blocks: FLINK-31816 Refactor EmbeddedLeaderElectionService
• causes: FLINK-32678 Release Testing: Stress-Test to cover multiple low-level changes in Flink
• causes: FLINK-32994 LeaderElectionDriver.toString() is not implemented
• causes: FLINK-32503 FLIP-285 technical debt
• Discovered while testing: FLINK-25235 Re-enable ZooKeeperLeaderElectionITCase#testJobExecutionOnClusterWithLeaderChange
• fixes: FLINK-30484 ZooKeeperLeaderElectionTest.testZooKeeperReelection timed out
• fixes: FLINK-30338 Clean leader election legacy code
• is duplicated by: FLINK-30338 Clean leader election legacy code
• requires: FLINK-25806 Remove legacy high availability services
• links to: GitHub Pull Request #21555
• mentioned in: Page Loading...

Sub-Tasks

• Merge DispatcherRunnerLeaderElectionLifecycleManager into DefaultDispatcherRunner: Resolved
• Remove JVM asserts from leader election code: Resolved
• Fix @GuardedBy and @Nullable annotations in DefaultLeaderElectionService: Resolved
• Make TestingLeaderElectionService comply to how the LeaderElectionService interface is intended: Closed
• LeaderElectionService.stop should always call revokeLeadership: Resolved
• DefaultMultipleComponentLeaderElectionService saves wrong leader session ID: Resolved
• DefaultMultipleComponentLeaderElectionService triggers HA backend change even if it's not the leader: Closed
• Refactor redundant code in AbstractHaServices: Resolved
• Migrate LeaderElection-related unit tests to JUnit5: Resolved
• Remove contender description from LeaderElectionDriverFactory interface: Resolved
• Separate DefaultLeaderElectionService.start(LeaderContender) into two separate methods for starting the driver and registering a contender: Resolved
• Introducing sub-interface LeaderElectionService.LeaderElection: Resolved
• Introduce contender ID into LeaderElectionService interface: Resolved
• Make DefaultLeaderElectionService implement MultipleComponentLeaderElectionService.Listener: Resolved
• Replace LeaderElectionDriver in DefaultLeaderElectionService with MultipleComponentLeaderElectionDriver: Resolved
• Add multiple-component support to DefaultLeaderElectionService: Resolved
• Move LeaderElectionService.stop() into LeaderElection.close(): Resolved
• Removing unused HighAvailabilityServices implementations: Resolved
• Move LeaderElectionService out of LeaderContender: Resolved
• Enable Precondition in DefaultLeaderElectionService.close after the MultipleComponentLeaderElectionDriverAdapter is removed: Resolved
• Move thread handling from DefaultMultipleComponentLeaderElectionService into DefaultLeaderElectionService: Resolved
• Remove unused KubernetesMultipleComponentLeaderElectionHaServicesFactory: Closed
• Moves DefaultLeaderElectionService.startLeaderElectionBackend() into HAServices: Closed
• Refactor MultipleComponentLeaderElectionDriver.Listener.notifyAllKnownLeaderInformation(Collection): Resolved
• Move error handling into MultipleComponentLeaderElectionDriverFactory: Resolved
• Replaces error handling functionality with onError method in MultipleComponentLeaderElectionDriver.Listener interface: Resolved
• Migrate KubernetesHighAvailabilityTestBase and implementing test classes: Resolved
• Migrate KubernetesLeaderElectionAndRetrievalITCase: Resolved
• Migrate ZooKeeperLeaderElectionTest: Resolved
• Moves componentId/contenderID handling from DefaultMultipleComponentLeaderElectionService into DefaultLeaderElectionService: Resolved
• Add fallback error handler to DefaultLeaderElectionService: Resolved
• Remove unused/obsolete classes: Resolved
• Renames contenderID into componentId: Resolved