[FLINK-21685] Flink JobManager failed to restart from checkpoint in kubernetes HA setup

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.12.1, 1.12.2
• Fix Version/s: 1.13.0, 1.12.3
• Component/s: Deployment / Kubernetes
• Labels: k8s-ha pull-request-available
• Assignee: Yang Wang
• Reporter: Peng Zhang
• Created: 09/Mar/21 07:22
• Updated: 28/Aug/21 11:13
• Resolved: 01/Apr/21 11:29

Description

We use Flink K8S session cluster with HA mode (1 JobManager and 4 TaskManagers). When jobs are running in Flink, and JobManager restarted, Flink JobManager failed to recover job from checkpoint

2021-03-08 13:16:42,962 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to fetch 1 checkpoints from storage. 
2021-03-08 13:16:42,962 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to fetch 1 checkpoints from storage. 
2021-03-08 13:16:42,962 INFO  org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to retrieve checkpoint 1. 
2021-03-08 13:16:43,014 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Restoring job 9a534b2e309b24f78866b65d94082ead from Checkpoint 1 @ 1615208258041 for 9a534b2e309b24f78866b65d94082ead located at s3a://zalando-stellar-flink-state-eu-central-1-staging/checkpoints/9a534b2e309b24f78866b65d94082ead/chk-1. 
2021-03-08 13:16:43,023 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No master state to restore 
2021-03-08 13:16:43,024 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@58d927d2 for BrandCollectionTrackingJob (9a534b2e309b24f78866b65d94082ead). 
2021-03-08 13:16:43,046 INFO  org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl      [] - JobManager runner for job BrandCollectionTrackingJob (9a534b2e309b24f78866b65d94082ead) was granted leadership with session id c258d8ce-69d3-49df-8bee-1b748d5bbe74 at akka.tcp://flink@10.2.179.12:6123/user/rpc/jobmanager_2. 
2021-03-08 13:16:43,060 WARN  akka.remote.transport.netty.NettyTransport                   [] - Remote connection to [null] failed with java.net.NoRouteToHostException: No route to host 
2021-03-08 13:16:43,060 WARN  akka.remote.ReliableDeliverySupervisor                       [] - Association with remote system [akka.tcp://flink@10.2.174.188:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@10.2.174.188:6123]] Caused by: [java.net.NoRouteToHostException: No route to host]

Attached is the log, and our configuration.

 

Issue Links

• duplicates: FLINK-21902 A deadlock while using K8s HA service
• relates to: FLINK-21382 Standalone K8s documentation does not explain usage of standby JobManagers
• links to: GitHub Pull Request #15385