[FLINK-24174] MiniClusterTestEnvironment‘s triggerTaskManagerFailover may stuck in CommonTestUtils.waitForJobStatus()

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Test Infrastructure
• Assignee: Unassigned
• Reporter: Liu
• Created: 07/Sep/21 02:30
• Updated: 07/Sep/21 03:04
• Resolved: 07/Sep/21 03:04

Description

When writing taskmanager failover tests with unified testing framework for connectors, I find that it may stuck in 

CommonTestUtils.waitForJobStatus() as following:

1. triggerTaskManagerFailover is called.
2. JobStatus switched from RUNNING to RESTARTING.
3. JobStatus switched from RESTARTING to RUNNING.
4. The method terminateTaskManager() is completed.
5. Since the jobStatus is RUNNING, CommonTestUtils.waitForJobStatus() will never exit.

A solution is to call terminateTaskManager() with async way. At the same time, call 

CommonTestUtils.waitForJobStatus(). The pseudo code can be as follow:

public void triggerTaskManagerFailover(JobClient jobClient, Runnable afterFailAction)
        throws Exception {
    CompletableFuture<Void> completableFuture = terminateTaskManager();
    CommonTestUtils.waitForJobStatus(
            jobClient,
            Arrays.asList(JobStatus.FAILING, JobStatus.FAILED, JobStatus.RESTARTING),
            Deadline.fromNow(Duration.ofMinutes(5)));
    completableFuture.get();
    afterFailAction.run();
    startTaskManager();
}

Issue Links

• duplicates: FLINK-23807 Use RestClient to detect restarts in MiniClusterTestEnvironment#triggerTaskManagerFailover