[FLINK-34343] ResourceManager registration is not completed when registering the JobMaster

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.17.2, 1.19.0, 1.18.1
• Fix Version/s: 1.19.0, 1.17.3, 1.18.2
• Component/s: Runtime / Coordination, Runtime / RPC
• Labels: pull-request-available test-stability
• Assignee: Chesnay Schepler
• Reporter: Matthias Pohl
• Created: 02/Feb/24 11:25
• Updated: 05/Feb/24 11:46
• Resolved: 05/Feb/24 11:46

Description

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57203&view=logs&j=64debf87-ecdb-5aef-788d-8720d341b5cb&t=2302fb98-0839-5df2-3354-bbae636f81a7&l=8066

The test run failed due to a NullPointerException:

Feb 02 01:11:55 2024-02-02 01:11:47,791 INFO  org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor       [] - The rpc endpoint org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager has not been started yet. Discarding message LocalFencedMessage(0000000000000000000
0000000000000, LocalRpcInvocation(ResourceManagerGateway.registerJobMaster(JobMasterId, ResourceID, String, JobID, Time))) until processing is started.
Feb 02 01:11:55 2024-02-02 01:11:47,797 WARN  org.apache.flink.runtime.rpc.pekko.SupervisorActor           [] - RpcActor pekko://flink/user/rpc/resourcemanager_2 has failed. Shutting it down now.
Feb 02 01:11:55 java.lang.NullPointerException: Cannot invoke "org.apache.flink.runtime.rpc.RpcServer.getAddress()" because "this.rpcServer" is null
Feb 02 01:11:55         at org.apache.flink.runtime.rpc.RpcEndpoint.getAddress(RpcEndpoint.java:322) ~[flink-dist-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:182) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253) ~[flink-rpc-akka06a9bb81-2e68-483a-b236-a283d0b1d097.jar:1.19-SNAPSHOT]
Feb 02 01:11:55         at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
Feb 02 01:11:55         at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) ~[?:?]
Feb 02 01:11:55         at java.util.concurrent.ForkJoinPool.scan(Unknown Source) ~[?:?]
Feb 02 01:11:55         at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[?:?]
Feb 02 01:11:55         at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[?:?]

Issue Links

• Discovered while testing: FLINK-34007 Flink Job stuck in suspend state after losing leadership in HA Mode
• links to: GitHub Pull Request #24258