[FLINK-21538] Elasticsearch6DynamicSinkITCase.testWritingDocuments fails when submitting job

• Type: Technical Debt
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12.1, 1.13.0
• Fix Version/s: 1.14.0, 1.13.3, 1.12.8
• Component/s: Connectors / ElasticSearch, Runtime / Coordination
• Labels: auto-deprioritized-major auto-unassigned pull-request-available test-stability
• Assignee: Till Rohrmann
• Reporter: Dawid Wysakowicz
• Created: 01/Mar/21 09:09
• Updated: 15/Dec/21 01:40
• Resolved: 23/Aug/21 09:50

Description

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13868&view=logs&j=3d12d40f-c62d-5ec4-6acc-0efe94cc3e89&t=5d6e4255-0ea8-5e2a-f52c-c881b7872361

2021-02-27T00:16:06.9493539Z org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-02-27T00:16:06.9494494Z 	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-02-27T00:16:06.9495733Z 	at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:117)
2021-02-27T00:16:06.9496596Z 	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-02-27T00:16:06.9497354Z 	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-02-27T00:16:06.9525795Z 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T00:16:06.9526744Z 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-02-27T00:16:06.9527784Z 	at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-02-27T00:16:06.9528552Z 	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-02-27T00:16:06.9529271Z 	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-02-27T00:16:06.9530013Z 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T00:16:06.9530482Z 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-02-27T00:16:06.9531068Z 	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1046)
2021-02-27T00:16:06.9531544Z 	at akka.dispatch.OnComplete.internal(Future.scala:264)
2021-02-27T00:16:06.9531908Z 	at akka.dispatch.OnComplete.internal(Future.scala:261)
2021-02-27T00:16:06.9532449Z 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-02-27T00:16:06.9532860Z 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-02-27T00:16:06.9533245Z 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2021-02-27T00:16:06.9533721Z 	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-02-27T00:16:06.9534225Z 	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2021-02-27T00:16:06.9534697Z 	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2021-02-27T00:16:06.9535217Z 	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2021-02-27T00:16:06.9535718Z 	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2021-02-27T00:16:06.9536127Z 	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:573)
2021-02-27T00:16:06.9536861Z 	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-02-27T00:16:06.9537394Z 	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-02-27T00:16:06.9537916Z 	at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2021-02-27T00:16:06.9605804Z 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2021-02-27T00:16:06.9606794Z 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2021-02-27T00:16:06.9607642Z 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2021-02-27T00:16:06.9608419Z 	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-02-27T00:16:06.9609252Z 	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
2021-02-27T00:16:06.9610024Z 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2021-02-27T00:16:06.9613676Z 	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2021-02-27T00:16:06.9615526Z 	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
2021-02-27T00:16:06.9616727Z 	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
2021-02-27T00:16:06.9617826Z 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
2021-02-27T00:16:06.9618940Z 	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2021-02-27T00:16:06.9620109Z 	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
2021-02-27T00:16:06.9621415Z 	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2021-02-27T00:16:06.9622598Z 	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2021-02-27T00:16:06.9623716Z Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2021-02-27T00:16:06.9625006Z 	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118)
2021-02-27T00:16:06.9626398Z 	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80)
2021-02-27T00:16:06.9628020Z 	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233)
2021-02-27T00:16:06.9629257Z 	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224)
2021-02-27T00:16:06.9630622Z 	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215)
2021-02-27T00:16:06.9631835Z 	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:669)
2021-02-27T00:16:06.9633415Z 	at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyTaskFailure(UpdateSchedulerNgOnInternalFailuresListener.java:56)
2021-02-27T00:16:06.9634940Z 	at org.apache.flink.runtime.executiongraph.ExecutionGraph.notifySchedulerNgAboutInternalTaskFailure(ExecutionGraph.java:1869)
2021-02-27T00:16:06.9636193Z 	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1437)
2021-02-27T00:16:06.9637220Z 	at org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:1377)
2021-02-27T00:16:06.9638462Z 	at org.apache.flink.runtime.executiongraph.Execution.markFailed(Execution.java:1205)
2021-02-27T00:16:06.9639683Z 	at org.apache.flink.runtime.executiongraph.Execution.lambda$deploy$11(Execution.java:856)
2021-02-27T00:16:06.9640771Z 	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-02-27T00:16:06.9641839Z 	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-02-27T00:16:06.9643554Z 	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
2021-02-27T00:16:06.9644658Z 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
2021-02-27T00:16:06.9645998Z 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
2021-02-27T00:16:06.9647143Z 	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
2021-02-27T00:16:06.9648506Z 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
2021-02-27T00:16:06.9649340Z 	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
2021-02-27T00:16:06.9650021Z 	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
2021-02-27T00:16:06.9650741Z 	at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2021-02-27T00:16:06.9651406Z 	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2021-02-27T00:16:06.9652093Z 	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
2021-02-27T00:16:06.9652972Z 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2021-02-27T00:16:06.9653685Z 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2021-02-27T00:16:06.9654385Z 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2021-02-27T00:16:06.9655010Z 	at akka.actor.Actor.aroundReceive(Actor.scala:517)
2021-02-27T00:16:06.9655606Z 	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
2021-02-27T00:16:06.9656223Z 	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
2021-02-27T00:16:06.9656910Z 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
2021-02-27T00:16:06.9739719Z 	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
2021-02-27T00:16:06.9740802Z 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
2021-02-27T00:16:06.9741521Z 	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
2021-02-27T00:16:06.9742086Z 	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
2021-02-27T00:16:06.9742776Z 	... 4 more
2021-02-27T00:16:06.9743982Z Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException: Invocation of public abstract java.util.concurrent.CompletableFuture org.apache.flink.runtime.taskexecutor.TaskExecutorGateway.submitTask(org.apache.flink.runtime.deployment.TaskDeploymentDescriptor,org.apache.flink.runtime.jobmaster.JobMasterId,org.apache.flink.api.common.time.Time) timed out.
2021-02-27T00:16:06.9745460Z 	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
2021-02-27T00:16:06.9746212Z 	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338)
2021-02-27T00:16:06.9746961Z 	at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925)
2021-02-27T00:16:06.9747806Z 	at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913)
2021-02-27T00:16:06.9748553Z 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T00:16:06.9749330Z 	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2021-02-27T00:16:06.9750210Z 	at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:234)
2021-02-27T00:16:06.9751031Z 	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-02-27T00:16:06.9751954Z 	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-02-27T00:16:06.9752836Z 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T00:16:06.9753594Z 	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
2021-02-27T00:16:06.9754400Z 	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1044)
2021-02-27T00:16:06.9755076Z 	at akka.dispatch.OnComplete.internal(Future.scala:263)
2021-02-27T00:16:06.9755623Z 	at akka.dispatch.OnComplete.internal(Future.scala:261)
2021-02-27T00:16:06.9756221Z 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-02-27T00:16:06.9756841Z 	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-02-27T00:16:06.9757772Z 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2021-02-27T00:16:06.9758524Z 	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-02-27T00:16:06.9759315Z 	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2021-02-27T00:16:06.9760053Z 	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2021-02-27T00:16:06.9760865Z 	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2021-02-27T00:16:06.9761785Z 	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2021-02-27T00:16:06.9762565Z 	at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650)
2021-02-27T00:16:06.9763213Z 	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
2021-02-27T00:16:06.9763902Z 	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
2021-02-27T00:16:06.9764625Z 	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
2021-02-27T00:16:06.9765323Z 	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
2021-02-27T00:16:06.9766035Z 	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
2021-02-27T00:16:06.9766812Z 	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
2021-02-27T00:16:06.9767768Z 	at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
2021-02-27T00:16:06.9768801Z 	at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
2021-02-27T00:16:06.9769614Z 	at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
2021-02-27T00:16:06.9770261Z 	at java.lang.Thread.run(Thread.java:748)
2021-02-27T00:16:06.9771578Z Caused by: java.util.concurrent.TimeoutException: Invocation of public abstract java.util.concurrent.CompletableFuture org.apache.flink.runtime.taskexecutor.TaskExecutorGateway.submitTask(org.apache.flink.runtime.deployment.TaskDeploymentDescriptor,org.apache.flink.runtime.jobmaster.JobMasterId,org.apache.flink.api.common.time.Time) timed out.
2021-02-27T00:16:06.9773056Z 	at org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway.submitTask(RpcTaskManagerGateway.java:68)
2021-02-27T00:16:06.9773869Z 	at org.apache.flink.runtime.executiongraph.Execution.lambda$deploy$10(Execution.java:832)
2021-02-27T00:16:06.9774669Z 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
2021-02-27T00:16:06.9775423Z 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2021-02-27T00:16:06.9776076Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-02-27T00:16:06.9776887Z 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2021-02-27T00:16:06.9777916Z 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2021-02-27T00:16:06.9778756Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-02-27T00:16:06.9779504Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-02-27T00:16:06.9780025Z 	... 1 more
2021-02-27T00:16:06.9782426Z Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/rpc/taskmanager_12#1583248815]] after [10000 ms]. Message of type [org.apache.flink.runtime.rpc.messages.LocalRpcInvocation]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.
2021-02-27T00:16:06.9783716Z 	at akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
2021-02-27T00:16:06.9784445Z 	at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650)
2021-02-27T00:16:06.9785102Z 	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
2021-02-27T00:16:06.9785957Z 	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
2021-02-27T00:16:06.9786701Z 	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
2021-02-27T00:16:06.9787410Z 	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
2021-02-27T00:16:06.9788182Z 	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
2021-02-27T00:16:06.9788982Z 	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
2021-02-27T00:16:06.9789866Z 	at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
2021-02-27T00:16:06.9790699Z 	at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
2021-02-27T00:16:06.9791644Z 	at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
2021-02-27T00:16:06.9792194Z 	... 1 more

Issue Links

• is related to: FLINK-23906 Increase akka.ask.timeout for tests using the MiniCluster
• relates to: FLINK-22333 Elasticsearch7DynamicSinkITCase.testWritingDocuments failed due to deploy task timeout.
• links to: GitHub Pull Request #16918