Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-26923

SavepointITCase.testStopWithSavepointFailsOverToSavepoint failed on azure

    XMLWordPrintableJSON

Details

    Description

      2022-03-29T05:50:18.9604940Z Mar 29 05:50:18 [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 28.306 s <<< FAILURE! - in org.apache.flink.test.checkpointing.SavepointITCase
2022-03-29T05:50:18.9609713Z Mar 29 05:50:18 [ERROR] org.apache.flink.test.checkpointing.SavepointITCase.testStopWithSavepointFailsOverToSavepoint  Time elapsed: 3.363 s  <<< ERROR!
2022-03-29T05:50:18.9611347Z Mar 29 05:50:18 org.apache.flink.util.FlinkException: Stop with savepoint operation could not be completed.
2022-03-29T05:50:18.9613057Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.StopWithSavepoint.onLeave(StopWithSavepoint.java:124)
2022-03-29T05:50:18.9614629Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.transitionToState(AdaptiveScheduler.java:1181)
2022-03-29T05:50:18.9616369Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.goToRestarting(AdaptiveScheduler.java:858)
2022-03-29T05:50:18.9618273Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.FailureResultUtil.restartOrFail(FailureResultUtil.java:28)
2022-03-29T05:50:18.9619815Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.StopWithSavepoint.onFailure(StopWithSavepoint.java:149)
2022-03-29T05:50:18.9621464Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.StateWithExecutionGraph.updateTaskExecutionState(StateWithExecutionGraph.java:367)
2022-03-29T05:50:18.9623122Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.lambda$updateTaskExecutionState$4(AdaptiveScheduler.java:496)
2022-03-29T05:50:18.9624528Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.State.tryCall(State.java:137)
2022-03-29T05:50:18.9626318Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.updateTaskExecutionState(AdaptiveScheduler.java:493)
2022-03-29T05:50:18.9627831Z Mar 29 05:50:18 	at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78)
2022-03-29T05:50:18.9629329Z Mar 29 05:50:18 	at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:443)
2022-03-29T05:50:18.9630643Z Mar 29 05:50:18 	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
2022-03-29T05:50:18.9632127Z Mar 29 05:50:18 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-03-29T05:50:18.9633394Z Mar 29 05:50:18 	at java.lang.reflect.Method.invoke(Method.java:498)
2022-03-29T05:50:18.9634943Z Mar 29 05:50:18 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:304)
2022-03-29T05:50:18.9636737Z Mar 29 05:50:18 	at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2022-03-29T05:50:18.9638234Z Mar 29 05:50:18 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:302)
2022-03-29T05:50:18.9639920Z Mar 29 05:50:18 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2022-03-29T05:50:18.9641506Z Mar 29 05:50:18 	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2022-03-29T05:50:18.9643007Z Mar 29 05:50:18 	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2022-03-29T05:50:18.9644379Z Mar 29 05:50:18 	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2022-03-29T05:50:18.9645829Z Mar 29 05:50:18 	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2022-03-29T05:50:18.9647316Z Mar 29 05:50:18 	at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2022-03-29T05:50:18.9648648Z Mar 29 05:50:18 	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2022-03-29T05:50:18.9650044Z Mar 29 05:50:18 	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2022-03-29T05:50:18.9651437Z Mar 29 05:50:18 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2022-03-29T05:50:18.9652830Z Mar 29 05:50:18 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2022-03-29T05:50:18.9654205Z Mar 29 05:50:18 	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2022-03-29T05:50:18.9655489Z Mar 29 05:50:18 	at akka.actor.Actor.aroundReceive(Actor.scala:537)
2022-03-29T05:50:18.9656976Z Mar 29 05:50:18 	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2022-03-29T05:50:18.9658156Z Mar 29 05:50:18 	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2022-03-29T05:50:18.9659677Z Mar 29 05:50:18 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2022-03-29T05:50:18.9660662Z Mar 29 05:50:18 	at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2022-03-29T05:50:18.9661444Z Mar 29 05:50:18 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2022-03-29T05:50:18.9662014Z Mar 29 05:50:18 	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2022-03-29T05:50:18.9663168Z Mar 29 05:50:18 	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2022-03-29T05:50:18.9668230Z Mar 29 05:50:18 	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
2022-03-29T05:50:18.9669554Z Mar 29 05:50:18 	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
2022-03-29T05:50:18.9670718Z Mar 29 05:50:18 	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
2022-03-29T05:50:18.9671952Z Mar 29 05:50:18 	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2022-03-29T05:50:18.9673252Z Mar 29 05:50:18 Caused by: org.apache.flink.runtime.operators.testutils.ExpectedTestException: Expected Test Exception
2022-03-29T05:50:18.9674967Z Mar 29 05:50:18 	at org.apache.flink.test.checkpointing.SavepointITCase$FailingOnCompletedSavepointMapFunction.notifyCheckpointComplete(SavepointITCase.java:367)
2022-03-29T05:50:18.9677056Z Mar 29 05:50:18 	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyCheckpointComplete(AbstractUdfStreamOperator.java:126)
2022-03-29T05:50:18.9678734Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:104)
2022-03-29T05:50:18.9680432Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.notifyCheckpointComplete(RegularOperatorChain.java:145)
2022-03-29T05:50:18.9682031Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpoint(SubtaskCheckpointCoordinatorImpl.java:409)
2022-03-29T05:50:18.9683810Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:343)
2022-03-29T05:50:18.9685416Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1421)
2022-03-29T05:50:18.9687234Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$17(StreamTask.java:1362)
2022-03-29T05:50:18.9689195Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$20(StreamTask.java:1401)
2022-03-29T05:50:18.9690889Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93)
2022-03-29T05:50:18.9692407Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
2022-03-29T05:50:18.9693862Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:362)
2022-03-29T05:50:18.9695483Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:348)
2022-03-29T05:50:18.9697230Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:225)
2022-03-29T05:50:18.9698683Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:871)
2022-03-29T05:50:18.9699947Z Mar 29 05:50:18 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:773)
2022-03-29T05:50:18.9701220Z Mar 29 05:50:18 	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
2022-03-29T05:50:18.9702487Z Mar 29 05:50:18 	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
2022-03-29T05:50:18.9703639Z Mar 29 05:50:18 	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
2022-03-29T05:50:18.9704731Z Mar 29 05:50:18 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
2022-03-29T05:50:18.9705865Z Mar 29 05:50:18 	at java.lang.Thread.run(Thread.java:748)
2022-03-29T05:50:18.9706814Z Mar 29 05:50:18

      https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33863&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=5661

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-27972 Race condition between task/savepoint notification failure

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #19281

          Activity

            People

              dwysakowicz Dawid Wysakowicz
              gaoyunhaii Yun Gao
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: