[FLINK-13601] RegionFailoverITCase is unstable

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.0, 1.10.0
• Fix Version/s: 1.10.0
• Component/s: Runtime / Checkpointing, Runtime / Coordination
• Labels: pull-request-available test-stability
• Assignee: Yun Tang
• Reporter: Aljoscha Krettek
• Created: 06/Aug/19 13:22
• Updated: 18/Oct/19 12:47
• Resolved: 18/Oct/19 12:47

Description

Excerpt from https://travis-ci.com/flink-ci/flink/jobs/222711830:

10:44:31.222 [INFO] Running org.apache.flink.test.checkpointing.RegionFailoverITCase
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: 9e0fbeaa580123e05cfce5554f443d23)
    at org.apache.flink.client.program.MiniClusterClient.submitJob(MiniClusterClient.java:92)
    at org.apache.flink.test.checkpointing.RegionFailoverITCase.testMultiRegionFailover(RegionFailoverITCase.java:132)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
    at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
    at org.apache.flink.client.program.MiniClusterClient.submitJob(MiniClusterClient.java:90)
    ... 13 more
Caused by: java.lang.RuntimeException: Test failed due to unexpected recovered index: 2000, while last completed checkpoint record index: 1837
    at org.apache.flink.test.checkpointing.RegionFailoverITCase$StringGeneratingSourceFunction.initializeState(RegionFailoverITCase.java:300)
    at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
    at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:862)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:367)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:688)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:518)
    ... 1 more
10:44:39.210 [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.983 s <<< FAILURE! - in org.apache.flink.test.checkpointing.RegionFailoverITCase

Issue Links

• is related to: FLINK-10712 RestartPipelinedRegionStrategy does not restore state
• links to: GitHub Pull Request #9386