[FLINK-25269] Return 503 Service Unavailable if endpoint is not ready yet

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / REST
• Labels: pull-request-available stale-assigned test-stability
• Release Note: The REST API now returns a 503 Service Unavailable error when a request is made but the backing component isn't ready yet. Previously this returned a 500 Internal Server error.
• Assignee: Chesnay Schepler
• Reporter: Yun Gao
• Created: 13/Dec/21 07:02
• Updated: 14/Apr/22 12:36
• Resolved: 14/Apr/22 12:36

Description

If an endpoint (e.g., the RM) has not been started yet when receiving a request from the REST API, then an exception is thrown that is treated as an unhandled exception. we can handle this in a nicer way by returning 503 Service Unavailable.

Dec 10 13:12:17 	at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_ae8961cc-571e-483d-b468-da652db90bb9.jar:1.15-SNAPSHOT]
Dec 10 13:12:17 	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_292]
Dec 10 13:12:17 	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_292]
Dec 10 13:12:17 	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_292]
Dec 10 13:12:17 	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_292]
Dec 10 13:12:17 2021-12-10 13:12:07,296 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Starting the resource manager.
Dec 10 13:12:17 2021-12-10 13:12:09,426 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 10.1.0.79:35275-d26a40 (akka.tcp://flink@10.1.0.79:35275/user/rpc/taskmanager_0) at ResourceManager
Dec 10 13:12:17 2021-12-10 13:12:09,496 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 10.1.0.79:35275-d26a40 (akka.tcp://flink@10.1.0.79:35275/user/rpc/taskmanager_0) at ResourceManager
Dec 10 13:12:17 2021-12-10 13:12:13,189 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Received JobGraph submission 'Elasticsearch7.x end to end sink test example' (effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,190 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Submitting job 'Elasticsearch7.x end to end sink test example' (effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,225 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_2 .
Dec 10 13:12:17 2021-12-10 13:12:13,252 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Initializing job 'Elasticsearch7.x end to end sink test example' (effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,324 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647, backoffTimeMS=1000) for Elasticsearch7.x end to end sink test example (effb86be1cfd961e947988200a724de7).
Dec 10 13:12:17 2021-12-10 13:12:13,428 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Running initialization on master for job Elasticsearch7.x end to end sink test example (effb86be1cfd961e947988200a724de7).
 

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27953&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=3408

Issue Links

• links to: GitHub Pull Request #19443