[FLINK-16373] EmbeddedLeaderService: IllegalStateException: The RPC connection is already closed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.9.3, 1.10.1, 1.11.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Robert Metzger
• Created: 02/Mar/20 10:36
• Updated: 02/Apr/20 09:01
• Resolved: 02/Apr/20 09:01

Description

In our CI system, I see a lot of these error messages:

09:52:41,108 [flink-akka.actor.default-dispatcher-2] INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Allocated slot for fd36e49b9105838fc7fb8fff442ade28.
09:52:41,108 [mini-cluster-io-thread-14] WARN  org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService  - Error notifying leader listener about new leader
java.lang.IllegalStateException: The RPC connection is already closed
	at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
	at org.apache.flink.runtime.registration.RegisteredRpcConnection.start(RegisteredRpcConnection.java:90)
	at org.apache.flink.runtime.taskexecutor.JobLeaderService$JobManagerLeaderListener.notifyLeaderAddress(JobLeaderService.java:334)
	at org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService$NotifyOfLeaderCall.run(EmbeddedLeaderService.java:515)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
09:52:41,109 [flink-akka.actor.default-dispatcher-4] INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Resolved JobManager address, beginning registration

Example cases

• https://transfer.sh/BAzbF/20200302.17.tar.gz
• https://transfer.sh/8344E/20200302.19.tar.gz

Issue Links

• links to: GitHub Pull Request #11313