[FLINK-22333] Elasticsearch7DynamicSinkITCase.testWritingDocuments failed due to deploy task timeout.

• Type: Technical Debt
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.14.0, 1.13.3, 1.12.8
• Component/s: Runtime / Coordination
• Labels: pull-request-available test-stability
• Assignee: Till Rohrmann
• Reporter: Guowei Ma
• Created: 18/Apr/21 02:48
• Updated: 15/Dec/21 01:40
• Resolved: 23/Aug/21 10:02

Description

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16694&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=03dca39c-73e8-5aaf-601d-328ae5c35f20&l=12329

2021-04-16T23:37:23.5719280Z Apr 16 23:37:23 org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-04-16T23:37:23.5739250Z Apr 16 23:37:23 	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-04-16T23:37:23.5759329Z Apr 16 23:37:23 	at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2021-04-16T23:37:23.5779145Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-04-16T23:37:23.5799204Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-04-16T23:37:23.5819302Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.5839106Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-16T23:37:23.5859276Z Apr 16 23:37:23 	at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-04-16T23:37:23.5868964Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-04-16T23:37:23.5869925Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-04-16T23:37:23.5919839Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.5959562Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-04-16T23:37:23.5989732Z Apr 16 23:37:23 	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081)
2021-04-16T23:37:23.6019422Z Apr 16 23:37:23 	at akka.dispatch.OnComplete.internal(Future.scala:264)
2021-04-16T23:37:23.6039067Z Apr 16 23:37:23 	at akka.dispatch.OnComplete.internal(Future.scala:261)
2021-04-16T23:37:23.6060126Z Apr 16 23:37:23 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-04-16T23:37:23.6089258Z Apr 16 23:37:23 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-04-16T23:37:23.6119150Z Apr 16 23:37:23 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-16T23:37:23.6139149Z Apr 16 23:37:23 	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-04-16T23:37:23.6159077Z Apr 16 23:37:23 	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-04-16T23:37:23.6189432Z Apr 16 23:37:23 	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-04-16T23:37:23.6215243Z Apr 16 23:37:23 	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
2021-04-16T23:37:23.6219148Z Apr 16 23:37:23 	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-04-16T23:37:23.6220221Z Apr 16 23:37:23 	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-04-16T23:37:23.6249411Z Apr 16 23:37:23 	at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
2021-04-16T23:37:23.6259145Z Apr 16 23:37:23 	at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
2021-04-16T23:37:23.6289272Z Apr 16 23:37:23 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-16T23:37:23.6309243Z Apr 16 23:37:23 	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-04-16T23:37:23.6359306Z Apr 16 23:37:23 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6369399Z Apr 16 23:37:23 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6389444Z Apr 16 23:37:23 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-04-16T23:37:23.6429180Z Apr 16 23:37:23 	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
2021-04-16T23:37:23.6449179Z Apr 16 23:37:23 	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
2021-04-16T23:37:23.6479350Z Apr 16 23:37:23 	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
2021-04-16T23:37:23.6509468Z Apr 16 23:37:23 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
2021-04-16T23:37:23.6529183Z Apr 16 23:37:23 	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2021-04-16T23:37:23.6549361Z Apr 16 23:37:23 	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
2021-04-16T23:37:23.6579428Z Apr 16 23:37:23 	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2021-04-16T23:37:23.6609544Z Apr 16 23:37:23 	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2021-04-16T23:37:23.6639433Z Apr 16 23:37:23 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2021-04-16T23:37:23.6689286Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2021-04-16T23:37:23.6709450Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2021-04-16T23:37:23.6739457Z Apr 16 23:37:23 	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:206)
2021-04-16T23:37:23.6769557Z Apr 16 23:37:23 	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:196)
2021-04-16T23:37:23.6799563Z Apr 16 23:37:23 	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:187)
2021-04-16T23:37:23.6819274Z Apr 16 23:37:23 	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:679)
2021-04-16T23:37:23.6849564Z Apr 16 23:37:23 	at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:51)
2021-04-16T23:37:23.6869382Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(DefaultExecutionGraph.java:1462)
2021-04-16T23:37:23.6899408Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1139)
2021-04-16T23:37:23.6909137Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1079)
2021-04-16T23:37:23.6929208Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:910)
2021-04-16T23:37:23.6949242Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.Execution.lambda$deploy$5(Execution.java:623)
2021-04-16T23:37:23.6979400Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-04-16T23:37:23.7009387Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-04-16T23:37:23.7039429Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
2021-04-16T23:37:23.7069410Z Apr 16 23:37:23 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
2021-04-16T23:37:23.7099512Z Apr 16 23:37:23 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
2021-04-16T23:37:23.7129490Z Apr 16 23:37:23 	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
2021-04-16T23:37:23.7159459Z Apr 16 23:37:23 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
2021-04-16T23:37:23.7179286Z Apr 16 23:37:23 	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
2021-04-16T23:37:23.7199284Z Apr 16 23:37:23 	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
2021-04-16T23:37:23.7219225Z Apr 16 23:37:23 	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
2021-04-16T23:37:23.7249358Z Apr 16 23:37:23 	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
2021-04-16T23:37:23.7279391Z Apr 16 23:37:23 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
2021-04-16T23:37:23.7309421Z Apr 16 23:37:23 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2021-04-16T23:37:23.7339402Z Apr 16 23:37:23 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2021-04-16T23:37:23.7369462Z Apr 16 23:37:23 	at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
2021-04-16T23:37:23.7389335Z Apr 16 23:37:23 	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
2021-04-16T23:37:23.7409312Z Apr 16 23:37:23 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
2021-04-16T23:37:23.7410391Z Apr 16 23:37:23 	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
2021-04-16T23:37:23.7429387Z Apr 16 23:37:23 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
2021-04-16T23:37:23.7430673Z Apr 16 23:37:23 	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
2021-04-16T23:37:23.7432413Z Apr 16 23:37:23 	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
2021-04-16T23:37:23.7432914Z Apr 16 23:37:23 	... 4 more
2021-04-16T23:37:23.7440249Z Apr 16 23:37:23 Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException: Invocation of public abstract java.util.concurrent.CompletableFuture org.apache.flink.runtime.taskexecutor.TaskExecutorGateway.submitTask(org.apache.flink.runtime.deployment.TaskDeploymentDescriptor,org.apache.flink.runtime.jobmaster.JobMasterId,org.apache.flink.api.common.time.Time) timed out.
2021-04-16T23:37:23.7460244Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
2021-04-16T23:37:23.7499472Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338)
2021-04-16T23:37:23.7566322Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925)
2021-04-16T23:37:23.7573573Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913)
2021-04-16T23:37:23.7574577Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.7575989Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2021-04-16T23:37:23.7604566Z Apr 16 23:37:23 	at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:234)
2021-04-16T23:37:23.7606034Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-04-16T23:37:23.7607770Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-04-16T23:37:23.7608801Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-04-16T23:37:23.7629547Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2021-04-16T23:37:23.7631134Z Apr 16 23:37:23 	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1079)
2021-04-16T23:37:23.7631919Z Apr 16 23:37:23 	at akka.dispatch.OnComplete.internal(Future.scala:263)
2021-04-16T23:37:23.7632614Z Apr 16 23:37:23 	at akka.dispatch.OnComplete.internal(Future.scala:261)
2021-04-16T23:37:23.7633406Z Apr 16 23:37:23 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-04-16T23:37:23.7634098Z Apr 16 23:37:23 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-04-16T23:37:23.7640769Z Apr 16 23:37:23 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-04-16T23:37:23.7641887Z Apr 16 23:37:23 	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-04-16T23:37:23.7642873Z Apr 16 23:37:23 	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-04-16T23:37:23.7644045Z Apr 16 23:37:23 	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-04-16T23:37:23.7645054Z Apr 16 23:37:23 	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:644)
2021-04-16T23:37:23.7645963Z Apr 16 23:37:23 	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
2021-04-16T23:37:23.7646872Z Apr 16 23:37:23 	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
2021-04-16T23:37:23.7647884Z Apr 16 23:37:23 	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
2021-04-16T23:37:23.7649065Z Apr 16 23:37:23 	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
2021-04-16T23:37:23.7650084Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
2021-04-16T23:37:23.7651191Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
2021-04-16T23:37:23.7652829Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
2021-04-16T23:37:23.7653995Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
2021-04-16T23:37:23.7654917Z Apr 16 23:37:23 	at java.lang.Thread.run(Thread.java:748)
2021-04-16T23:37:23.7656367Z Apr 16 23:37:23 Caused by: java.util.concurrent.TimeoutException: Invocation of public abstract java.util.concurrent.CompletableFuture org.apache.flink.runtime.taskexecutor.TaskExecutorGateway.submitTask(org.apache.flink.runtime.deployment.TaskDeploymentDescriptor,org.apache.flink.runtime.jobmaster.JobMasterId,org.apache.flink.api.common.time.Time) timed out.
2021-04-16T23:37:23.7657965Z Apr 16 23:37:23 	at org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway.submitTask(RpcTaskManagerGateway.java:60)
2021-04-16T23:37:23.7660327Z Apr 16 23:37:23 	at org.apache.flink.runtime.executiongraph.Execution.lambda$deploy$4(Execution.java:599)
2021-04-16T23:37:23.7661402Z Apr 16 23:37:23 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
2021-04-16T23:37:23.7662383Z Apr 16 23:37:23 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2021-04-16T23:37:23.7663486Z Apr 16 23:37:23 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-04-16T23:37:23.7664487Z Apr 16 23:37:23 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2021-04-16T23:37:23.7665672Z Apr 16 23:37:23 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2021-04-16T23:37:23.7667016Z Apr 16 23:37:23 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-04-16T23:37:23.7668000Z Apr 16 23:37:23 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-04-16T23:37:23.7669323Z Apr 16 23:37:23 	... 1 more
2021-04-16T23:37:23.7671437Z Apr 16 23:37:23 Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/rpc/taskmanager_12#-550158139]] after [10000 ms]. Message of type [org.apache.flink.runtime.rpc.messages.LocalRpcInvocation]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.
2021-04-16T23:37:23.7672812Z Apr 16 23:37:23 	at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
2021-04-16T23:37:23.7673641Z Apr 16 23:37:23 	at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
2021-04-16T23:37:23.7674306Z Apr 16 23:37:23 	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
2021-04-16T23:37:23.7674893Z Apr 16 23:37:23 	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
2021-04-16T23:37:23.7675557Z Apr 16 23:37:23 	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
2021-04-16T23:37:23.7676277Z Apr 16 23:37:23 	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
2021-04-16T23:37:23.7676956Z Apr 16 23:37:23 	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
2021-04-16T23:37:23.7677706Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
2021-04-16T23:37:23.7678659Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
2021-04-16T23:37:23.7761875Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
2021-04-16T23:37:23.7763911Z Apr 16 23:37:23 	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
2021-04-16T23:37:23.7764773Z Apr 16 23:37:23 	... 1 more

Issue Links

• is related to: FLINK-23906 Increase akka.ask.timeout for tests using the MiniCluster
• is related to: FLINK-21538 Elasticsearch6DynamicSinkITCase.testWritingDocuments fails when submitting job
• links to: GitHub Pull Request #16924