Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21600

Resuming Savepoint (rocks, no parallelism change, heap timers) end-to-end test failed

    XMLWordPrintableJSON

    Details

      Description

      https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14081&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610

      2021-03-03T23:23:22.9286204Z  The program finished with the following exception:
      2021-03-03T23:23:22.9286551Z
      2021-03-03T23:23:22.9287394Z org.apache.flink.util.FlinkException: Could not stop with a savepoint job "6c88aa43b703192e88987473e722c22c".
      2021-03-03T23:23:22.9288135Z    at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
      2021-03-03T23:23:22.9288846Z    at org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
      2021-03-03T23:23:22.9289787Z    at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
      2021-03-03T23:23:22.9290721Z    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
      2021-03-03T23:23:22.9291800Z    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
      2021-03-03T23:23:22.9292565Z    at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
      2021-03-03T23:23:22.9293368Z    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
      2021-03-03T23:23:22.9294111Z Caused by: java.util.concurrent.TimeoutException
      2021-03-03T23:23:22.9294842Z    at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
      2021-03-03T23:23:22.9295735Z    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
      2021-03-03T23:23:22.9296418Z    at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579)
      2021-03-03T23:23:22.9297527Z    ... 6 more
      2021-03-03T23:23:22.9622887Z Mar 03 23:23:22 Waiting for job (6c88aa43b703192e88987473e722c22c) to reach terminal state FINISHED ...
      2021-03-03T23:36:57.3094695Z Mar 03 23:36:57 Test (pid: 3517) did not finish after 900 seconds.
      

      killing logs

      2021-03-03T23:36:57.3232128Z Mar 03 23:36:57 java.util.concurrent.RejectedExecutionException: event executor terminated
      2021-03-03T23:36:57.3233348Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3234766Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3236170Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3237567Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3265526Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3267472Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:471) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3268806Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3275491Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3277092Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3279030Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3280277Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3281591Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:139) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3282748Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:123) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3283868Z Mar 03 23:36:57    at org.apache.flink.runtime.rest.RestClient.submitRequest(RestClient.java:421) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3284969Z Mar 03 23:36:57    at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:344) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3286053Z Mar 03 23:36:57    at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:258) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3287244Z Mar 03 23:36:57    at org.apache.flink.client.program.rest.RestClusterClient.lambda$sendRetriableRequest$23(RestClusterClient.java:777) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3288091Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) [?:?]
      2021-03-03T23:36:57.3288763Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
      2021-03-03T23:36:57.3289660Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610) [?:?]
      2021-03-03T23:36:57.3290339Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649) [?:?]
      2021-03-03T23:36:57.3291011Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
      2021-03-03T23:36:57.3291678Z Mar 03 23:36:57    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      2021-03-03T23:36:57.3292340Z Mar 03 23:36:57    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      2021-03-03T23:36:57.3292927Z Mar 03 23:36:57    at java.lang.Thread.run(Thread.java:834) [?:?]
      2021-03-03T23:36:57.3294023Z Mar 03 23:36:57 2021-03-03 23:23:22,926 ERROR org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.rejectedExecution [] - Failed to submit a listener notification task. Event loop shut down?
      2021-03-03T23:36:57.3294797Z Mar 03 23:36:57 java.util.concurrent.RejectedExecutionException: event executor terminated
      2021-03-03T23:36:57.3330282Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3332091Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3333454Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3334983Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3336322Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3337809Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3339118Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3368256Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3369794Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3371107Z Mar 03 23:36:57    at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3372302Z Mar 03 23:36:57    at org.apache.flink.runtime.rest.RestClient.submitRequest(RestClient.java:425) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3373399Z Mar 03 23:36:57    at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:344) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3374497Z Mar 03 23:36:57    at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:258) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3375694Z Mar 03 23:36:57    at org.apache.flink.client.program.rest.RestClusterClient.lambda$sendRetriableRequest$23(RestClusterClient.java:777) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3376535Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) [?:?]
      2021-03-03T23:36:57.3377218Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
      2021-03-03T23:36:57.3377874Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610) [?:?]
      2021-03-03T23:36:57.3378745Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649) [?:?]
      2021-03-03T23:36:57.3379443Z Mar 03 23:36:57    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
      2021-03-03T23:36:57.3380305Z Mar 03 23:36:57    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      2021-03-03T23:36:57.3381397Z Mar 03 23:36:57    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      2021-03-03T23:36:57.3382019Z Mar 03 23:36:57    at java.lang.Thread.run(Thread.java:834) [?:?]
      2021-03-03T23:36:57.3382989Z Mar 03 23:36:57 2021-03-03 23:23:22,926 ERROR org.apache.flink.client.cli.CliFrontend                      [] - Error while running the command.
      2021-03-03T23:36:57.3383776Z Mar 03 23:36:57 org.apache.flink.util.FlinkException: Could not stop with a savepoint job "6c88aa43b703192e88987473e722c22c".
      2021-03-03T23:36:57.3385129Z Mar 03 23:36:57    at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3422634Z Mar 03 23:36:57    at org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3423852Z Mar 03 23:36:57    at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      2021-03-03T23:36:57.3425311Z Mar 03 23:36:57    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              maguowei Guowei Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: