Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11134

Invalid REST API request should not log the full exception in Flink logs

    XMLWordPrintableJSON

    Details

      Description

      2018-12-11 17:52:19,207 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler - Exception occurred in REST handler.
      org.apache.flink.runtime.rest.NotFoundException: Job 15d06690e88d309aa1bdbb6ce7c6dcd1 not found
      at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$1(AbstractExecutionGraphHandler.java:90)
      at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
      at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      at org.apache.flink.runtime.rest.handler.legacy.ExecutionGraphCache.lambda$getExecutionGraph$0(ExecutionGraphCache.java:133)
      at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
      at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
      at akka.dispatch.OnComplete.internal(Future.scala:258)
      at akka.dispatch.OnComplete.internal(Future.scala:256)
      at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
      at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:20)
      at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:18)
      at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
      at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
      at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
      at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
      at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
      at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
      at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
      at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
      at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
      Caused by: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job (15d06690e88d309aa1bdbb6ce7c6dcd1)
      at org.apache.flink.runtime.dispatcher.Dispatcher.getJobMasterGatewayFuture(Dispatcher.java:766)
      at org.apache.flink.runtime.dispatcher.Dispatcher.requestJob(Dispatcher.java:485)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
      at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
      at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
      at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
      at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
      at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
      at akka.actor.ActorCell.invoke(ActorCell.scala:495)
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
      at akka.dispatch.Mailbox.run(Mailbox.scala:224)
      at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
      ... 4 more
      

      Above stack trace gets logged when an exception occurs in the REST API due to a bad user request.

      In this case, a user had a browser tab open on Flink Web UI with a job details page. While this page is open, browser polls the REST API every couple seconds to get the status of the job. However, the existing job was replaced by another job (with a different ID) while this page continues to poll the API with an outdated job ID. 

      This should return a HTTP response code of 400 but it shouldn't log the exception stack trace in the Flink logs. This can cause the logs to get polluted real quick if someone has a Flink Web UI tab open somewhere with the outdated job ID.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chesnay Chesnay Schepler
                Reporter:
                markcho Mark Cho
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m