Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10314

Blocking calls in Execution Graph creation bring down cluster

    Details

      Description

      When the execution graph creation has long calls (or blocking calls), it brings down the cluster. It seems to block the main actor thread of the dispatcher, making the entire cluster unresponsive (web UI, etc.)

      Reproducing

      One way to reproduce this is to use a FileSystem that blocks on initialization or on opening a stream.

      Full Exception Log

      at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:1022)
      	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:985)
      	at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:159)
      	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
      	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
      	at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
      	at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:61)
      	at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:443)
      	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:248)
      	at org.apache.flink.runtime.executiongraph.ExecutionGraph.enableCheckpointing(ExecutionGraph.java:495)
      	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:345)
      	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
      	at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1152)
      	at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1132)
      	at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:295)
      	at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
      	at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:936)
      	at org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:291)
      	at org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:281)
      	at org.apache.flink.runtime.dispatcher.Dispatcher.persistAndRunJob(Dispatcher.java:266)
      	at org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$108/1800332159.acceptWithException(Unknown Source)
      	at org.apache.flink.util.function.ConsumerWithException.accept(ConsumerWithException.java:38)
      	at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$waitForTerminatingJobManager$29(Dispatcher.java:820)
      	at org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$110/1318857678.run(Unknown Source)
      	at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705)
      	at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:687)
      	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
      
      
      As a result, everything becomes unresponsive and breaks, and log is full of these errors:
      
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:06:58,679 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:08,629 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:08,663 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:08,679 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:08,700 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:18,712 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:18,729 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:28,739 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:28,749 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:38,709 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:38,729 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:38,769 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:38,778 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:48,788 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      2018-08-24 17:07:48,799 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler  - Could not retrieve the redirect address.
      java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
      	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
      	at akka.dispatch.OnComplete.internal(Future.scala:258)
      	at akka.dispatch.OnComplete.internal(Future.scala:256)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
      	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
      	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
      	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
      	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
      	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
      	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#146546706]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
      	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      	... 9 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                till.rohrmann Till Rohrmann
                Reporter:
                StephanEwen Stephan Ewen
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: