[FLINK-25893] ResourceManagerServiceImpl's lifecycle can lead to exceptions

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.14.3, 1.15.0
• Fix Version/s: 1.14.4, 1.15.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Xintong Song
• Reporter: Till Rohrmann
• Created: 31/Jan/22 10:22
• Updated: 21/Feb/22 10:56
• Resolved: 21/Feb/22 10:55

Description

The ResourceManagerServiceImpl lifecycle can lead to exceptions when calling ResourceManagerServiceImpl.deregisterApplication. The problem arises when the DispatcherResourceManagerComponent is shutdown before the ResourceManagerServiceImpl gains leadership or while it is starting the ResourceManager.

One problem is that deregisterApplication returns an exceptionally completed future if there is no leading ResourceManager.

Another problem is that if there is a leading ResourceManager, then it can still be the case that it has not been started yet. If this is the case, then ResourceManagerGateway.deregisterApplication will be discarded. The reason for this behaviour is that we create a ResourceManager in one Runnable and only start it in another. Due to this there can be the deregisterApplication call that gets the lock in between.

I'd suggest to correct the lifecycle and contract of the ResourceManagerServiceImpl.deregisterApplication.

Please note that due to this problem, the error reporting of this method has been suppressed. See FLINK-25885 for more details.

Issue Links

• is related to: FLINK-23240 ResumeCheckpointManuallyITCase.testExternalizedFSCheckpointsWithLocalRecoveryZookeeper fails on azure
• is related to: FLINK-25885 ClusterEntrypointTest.testWorkingDirectoryIsDeletedIfApplicationCompletes failed on azure
• links to: GitHub Pull Request #18567
• links to: GitHub Pull Request #18853
• links to: GitHub Pull Request #18856