[FLINK-30732] "Running HA (hashmap, sync) end-to-end test" failed with exit code 1

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.15.3
• Component/s: Tests
• Labels: auto-deprioritized-critical test-stability
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 18/Jan/23 08:50
• Updated: 19/Aug/23 22:35

Description

"Running HA (hashmap, sync) end-to-end test" failed in the following build:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44988&view=logs&j=e9d3d34f-3d15-59f4-0e3e-35067d100dfe&t=f8a6d3eb-38cf-5cca-9a99-d0badeb5fe62&l=13253

Some initial investigation revealed that there might have been problems with the TaskManager availability? I found a reoccuring stacktrace about a slot allocation failure:

2023-01-18 02:17:08,677 WARN  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Slot allocation for slot 127.0.0.1:43245-83194d_1 for job 3d663afea41cc2fdec0e2c91b671f2bc failed.
org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException: Could not send message [RemoteRpcInvocation(TaskExecutorGateway.requestSlot(SlotID, JobID, AllocationID, ResourceProfile, String, ResourceManagerId, Time))] from sender [Actor[akka://flink/temp/taskmanager_0$oh]] to recipient [Actor[akka.tcp://flink@127.0.0.1:43245/user/rpc/taskmanager_0#1526007138]], because the recipient is unreachable. This can either mean that the recipient has been terminated or that the remote RpcService is currently not reachable.
        at org.apache.flink.runtime.rpc.akka.DeadLettersActor.handleDeadLetter(DeadLettersActor.java:61) ~[flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_a2e10295-0436-48e6-a879-dfe2b8967a56.jar:1.15-SNAPSHOT]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
        at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]

Issue Links

• is related to: FLINK-23210 Running HA per-job cluster (hashmap, sync) end-to-end test failed on azure
• relates to: FLINK-30909 Running HA (hashmap, sync) end-to-end test