Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21329

"Local recovery and sticky scheduling end-to-end test" does not finish within 600 seconds

Details

    Description

      https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13118&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529&l=38515

      Feb 08 22:25:46 ==============================================================================
      Feb 08 22:25:46 Running 'Local recovery and sticky scheduling end-to-end test'
      Feb 08 22:25:46 ==============================================================================
      Feb 08 22:25:46 TEST_DATA_DIR: /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-46881214821
      Feb 08 22:25:47 Flink dist directory: /home/vsts/work/1/s/flink-dist/target/flink-1.13-SNAPSHOT-bin/flink-1.13-SNAPSHOT
      Feb 08 22:25:47 Running local recovery test with configuration:
      Feb 08 22:25:47         parallelism: 4
      Feb 08 22:25:47         max attempts: 10
      Feb 08 22:25:47         backend: rocks
      Feb 08 22:25:47         incremental checkpoints: false
      Feb 08 22:25:47         kill JVM: false
      Feb 08 22:25:47 Starting zookeeper daemon on host fv-az127-394.
      Feb 08 22:25:47 Starting HA cluster with 1 masters.
      Feb 08 22:25:48 Starting standalonesession daemon on host fv-az127-394.
      Feb 08 22:25:49 Starting taskexecutor daemon on host fv-az127-394.
      Feb 08 22:25:49 Waiting for Dispatcher REST endpoint to come up...
      Feb 08 22:25:50 Waiting for Dispatcher REST endpoint to come up...
      Feb 08 22:25:51 Waiting for Dispatcher REST endpoint to come up...
      Feb 08 22:25:53 Waiting for Dispatcher REST endpoint to come up...
      Feb 08 22:25:54 Dispatcher REST endpoint is up.
      Feb 08 22:25:54 Started TM watchdog with PID 28961.
      Feb 08 22:25:58 Job has been submitted with JobID e790e85a39040539f9386c0df7ca4812
      Feb 08 22:35:47 Test (pid: 27970) did not finish after 600 seconds.
      Feb 08 22:35:47 Printing Flink logs and killing it:
      
      

      and

      	at org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver.unhandledError(ZooKeeperLeaderRetrievalDriver.java:184)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:713)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:709)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
      	at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:708)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:874)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
      	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
      	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862)
      	... 10 more
      
      

      Attachments

        Issue Links

          Activity

            Similar outcome: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13326&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610

            2021-02-15T01:02:52.3770907Z ------------------------------------------------------------
            2021-02-15T01:02:52.3772409Z  The program finished with the following exception:
            2021-02-15T01:02:52.3773214Z 
            2021-02-15T01:02:52.3774604Z org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.util.concurrent.TimeoutException
            2021-02-15T01:02:52.3775820Z 	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
            2021-02-15T01:02:52.3776889Z 	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
            2021-02-15T01:02:52.3778111Z 	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
            2021-02-15T01:02:52.3778824Z 	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
            2021-02-15T01:02:52.3779712Z 	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
            2021-02-15T01:02:52.3780328Z 	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
            2021-02-15T01:02:52.3780963Z 	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
            2021-02-15T01:02:52.3781663Z 	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
            2021-02-15T01:02:52.3782340Z 	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
            2021-02-15T01:02:52.3782958Z Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
            2021-02-15T01:02:52.3783619Z 	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
            2021-02-15T01:02:52.3784257Z 	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
            2021-02-15T01:02:52.3784973Z 	at org.apache.flink.client.program.StreamContextEnvironment.getJobExecutionResult(StreamContextEnvironment.java:123)
            2021-02-15T01:02:52.3785740Z 	at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:80)
            2021-02-15T01:02:52.3786561Z 	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1782)
            2021-02-15T01:02:52.3787396Z 	at org.apache.flink.streaming.tests.StickyAllocationAndLocalRecoveryTestJob.main(StickyAllocationAndLocalRecoveryTestJob.java:138)
            2021-02-15T01:02:52.3788110Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            2021-02-15T01:02:52.3788757Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            2021-02-15T01:02:52.3791150Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            2021-02-15T01:02:52.3791827Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            2021-02-15T01:02:52.3792706Z 	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
            2021-02-15T01:02:52.3793248Z 	... 8 more
            2021-02-15T01:02:52.3793651Z Caused by: java.util.concurrent.TimeoutException
            2021-02-15T01:02:52.3794206Z 	at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1220)
            2021-02-15T01:02:52.3794897Z 	at org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
            2021-02-15T01:02:52.3795593Z 	at org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$15(FutureUtils.java:582)
            2021-02-15T01:02:52.3796266Z 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            2021-02-15T01:02:52.3796880Z 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            2021-02-15T01:02:52.3797574Z 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
            2021-02-15T01:02:52.3798805Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            2021-02-15T01:02:52.3800632Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            2021-02-15T01:02:52.3801242Z 	at java.base/java.lang.Thread.run(Thread.java:834)
            2021-02-15T01:54:04.9335391Z ==========================================================================================
            2021-02-15T01:54:04.9336579Z === WARNING: This E2E Run took already 80% of the allocated time budget of 250 minutes ===
            2021-02-15T01:54:04.9338033Z ==========================================================================================
            2021-02-15T02:33:04.9292547Z ======================================================================================================
            2021-02-15T02:33:04.9293721Z === WARNING: This E2E Run will time out in the next few minutes. Starting to upload the log output ===
            2021-02-15T02:33:04.9294789Z ======================================================================================================
            
            dwysakowicz Dawid Wysakowicz added a comment - Similar outcome: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13326&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610 2021-02-15T01:02:52.3770907Z ------------------------------------------------------------ 2021-02-15T01:02:52.3772409Z The program finished with the following exception: 2021-02-15T01:02:52.3773214Z 2021-02-15T01:02:52.3774604Z org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.util.concurrent.TimeoutException 2021-02-15T01:02:52.3775820Z at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366) 2021-02-15T01:02:52.3776889Z at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219) 2021-02-15T01:02:52.3778111Z at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) 2021-02-15T01:02:52.3778824Z at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) 2021-02-15T01:02:52.3779712Z at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) 2021-02-15T01:02:52.3780328Z at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) 2021-02-15T01:02:52.3780963Z at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) 2021-02-15T01:02:52.3781663Z at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) 2021-02-15T01:02:52.3782340Z at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) 2021-02-15T01:02:52.3782958Z Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException 2021-02-15T01:02:52.3783619Z at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) 2021-02-15T01:02:52.3784257Z at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) 2021-02-15T01:02:52.3784973Z at org.apache.flink.client.program.StreamContextEnvironment.getJobExecutionResult(StreamContextEnvironment.java:123) 2021-02-15T01:02:52.3785740Z at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:80) 2021-02-15T01:02:52.3786561Z at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1782) 2021-02-15T01:02:52.3787396Z at org.apache.flink.streaming.tests.StickyAllocationAndLocalRecoveryTestJob.main(StickyAllocationAndLocalRecoveryTestJob.java:138) 2021-02-15T01:02:52.3788110Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2021-02-15T01:02:52.3788757Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2021-02-15T01:02:52.3791150Z at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2021-02-15T01:02:52.3791827Z at java.base/java.lang.reflect.Method.invoke(Method.java:566) 2021-02-15T01:02:52.3792706Z at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349) 2021-02-15T01:02:52.3793248Z ... 8 more 2021-02-15T01:02:52.3793651Z Caused by: java.util.concurrent.TimeoutException 2021-02-15T01:02:52.3794206Z at org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1220) 2021-02-15T01:02:52.3794897Z at org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217) 2021-02-15T01:02:52.3795593Z at org.apache.flink.runtime.concurrent.FutureUtils.lambda$orTimeout$15(FutureUtils.java:582) 2021-02-15T01:02:52.3796266Z at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 2021-02-15T01:02:52.3796880Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2021-02-15T01:02:52.3797574Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 2021-02-15T01:02:52.3798805Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2021-02-15T01:02:52.3800632Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2021-02-15T01:02:52.3801242Z at java.base/java.lang. Thread .run( Thread .java:834) 2021-02-15T01:54:04.9335391Z ========================================================================================== 2021-02-15T01:54:04.9336579Z === WARNING: This E2E Run took already 80% of the allocated time budget of 250 minutes === 2021-02-15T01:54:04.9338033Z ========================================================================================== 2021-02-15T02:33:04.9292547Z ====================================================================================================== 2021-02-15T02:33:04.9293721Z === WARNING: This E2E Run will time out in the next few minutes. Starting to upload the log output === 2021-02-15T02:33:04.9294789Z ======================================================================================================
            maguowei Guowei Ma added a comment -

            https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14422&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610

            2021-03-10T23:56:19.2456160Z  The program finished with the following exception:
            2021-03-10T23:56:19.2456749Z 
            2021-03-10T23:56:19.2469726Z org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background operation retry gave up
            2021-03-10T23:56:19.2472547Z 	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
            2021-03-10T23:56:19.2473760Z 	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
            2021-03-10T23:56:19.2474822Z 	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
            2021-03-10T23:56:19.2476172Z 	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
            2021-03-10T23:56:19.2477146Z 	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
            2021-03-10T23:56:19.2478086Z 	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
            2021-03-10T23:56:19.2479065Z 	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
            2021-03-10T23:56:19.2480129Z 	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
            2021-03-10T23:56:19.2486127Z 	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
            2021-03-10T23:56:19.2487084Z Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background operation retry gave up
            2021-03-10T23:56:19.2487873Z 	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
            2021-03-10T23:56:19.2488470Z 	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
            2021-03-10T23:56:19.2489106Z 	at org.apache.flink.client.program.StreamContextEnvironment.getJobExecutionResult(StreamContextEnvironment.java:123)
            2021-03-10T23:56:19.2489773Z 	at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:80)
            2021-03-10T23:56:19.2490443Z 	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1839)
            2021-03-10T23:56:19.2491167Z 	at org.apache.flink.streaming.tests.StickyAllocationAndLocalRecoveryTestJob.main(StickyAllocationAndLocalRecoveryTestJob.java:139)
            2021-03-10T23:56:19.2491807Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            2021-03-10T23:56:19.2492391Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            2021-03-10T23:56:19.2493322Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            2021-03-10T23:56:19.2494397Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            2021-03-10T23:56:19.2494976Z 	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
            2021-03-10T23:56:19.2495419Z 	... 8 more
            2021-03-10T23:56:19.2495958Z Caused by: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background operation retry gave up
            2021-03-10T23:56:19.2496708Z 	at org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver.unhandledError(ZooKeeperLeaderRetrievalDriver.java:184)
            2021-03-10T23:56:19.2497400Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:713)
            2021-03-10T23:56:19.2498080Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:709)
            2021-03-10T23:56:19.2498749Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
            2021-03-10T23:56:19.2499441Z 	at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
            2021-03-10T23:56:19.2500135Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92)
            2021-03-10T23:56:19.2500932Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:708)
            2021-03-10T23:56:19.2501636Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:874)
            2021-03-10T23:56:19.2502420Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990)
            2021-03-10T23:56:19.2503156Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
            2021-03-10T23:56:19.2503866Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
            2021-03-10T23:56:19.2504599Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
            2021-03-10T23:56:19.2505190Z 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            2021-03-10T23:56:19.2505985Z 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
            2021-03-10T23:56:19.2506603Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            2021-03-10T23:56:19.2507161Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            2021-03-10T23:56:19.2507654Z 	at java.base/java.lang.Thread.run(Thread.java:834)
            2021-03-10T23:56:19.2508336Z Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
            2021-03-10T23:56:19.2508994Z 	at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
            2021-03-10T23:56:19.2509662Z 	at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862)
            2021-03-10T23:56:19.2510188Z 	... 9 more
            
            maguowei Guowei Ma added a comment - https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14422&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610 2021-03-10T23:56:19.2456160Z The program finished with the following exception: 2021-03-10T23:56:19.2456749Z 2021-03-10T23:56:19.2469726Z org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background operation retry gave up 2021-03-10T23:56:19.2472547Z at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) 2021-03-10T23:56:19.2473760Z at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) 2021-03-10T23:56:19.2474822Z at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) 2021-03-10T23:56:19.2476172Z at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) 2021-03-10T23:56:19.2477146Z at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) 2021-03-10T23:56:19.2478086Z at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) 2021-03-10T23:56:19.2479065Z at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) 2021-03-10T23:56:19.2480129Z at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) 2021-03-10T23:56:19.2486127Z at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) 2021-03-10T23:56:19.2487084Z Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background operation retry gave up 2021-03-10T23:56:19.2487873Z at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) 2021-03-10T23:56:19.2488470Z at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) 2021-03-10T23:56:19.2489106Z at org.apache.flink.client.program.StreamContextEnvironment.getJobExecutionResult(StreamContextEnvironment.java:123) 2021-03-10T23:56:19.2489773Z at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:80) 2021-03-10T23:56:19.2490443Z at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1839) 2021-03-10T23:56:19.2491167Z at org.apache.flink.streaming.tests.StickyAllocationAndLocalRecoveryTestJob.main(StickyAllocationAndLocalRecoveryTestJob.java:139) 2021-03-10T23:56:19.2491807Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2021-03-10T23:56:19.2492391Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2021-03-10T23:56:19.2493322Z at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2021-03-10T23:56:19.2494397Z at java.base/java.lang.reflect.Method.invoke(Method.java:566) 2021-03-10T23:56:19.2494976Z at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) 2021-03-10T23:56:19.2495419Z ... 8 more 2021-03-10T23:56:19.2495958Z Caused by: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background operation retry gave up 2021-03-10T23:56:19.2496708Z at org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver.unhandledError(ZooKeeperLeaderRetrievalDriver.java:184) 2021-03-10T23:56:19.2497400Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:713) 2021-03-10T23:56:19.2498080Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:709) 2021-03-10T23:56:19.2498749Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) 2021-03-10T23:56:19.2499441Z at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) 2021-03-10T23:56:19.2500135Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) 2021-03-10T23:56:19.2500932Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:708) 2021-03-10T23:56:19.2501636Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:874) 2021-03-10T23:56:19.2502420Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) 2021-03-10T23:56:19.2503156Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) 2021-03-10T23:56:19.2503866Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) 2021-03-10T23:56:19.2504599Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) 2021-03-10T23:56:19.2505190Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 2021-03-10T23:56:19.2505985Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 2021-03-10T23:56:19.2506603Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 2021-03-10T23:56:19.2507161Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 2021-03-10T23:56:19.2507654Z at java.base/java.lang. Thread .run( Thread .java:834) 2021-03-10T23:56:19.2508336Z Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss 2021-03-10T23:56:19.2508994Z at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:102) 2021-03-10T23:56:19.2509662Z at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) 2021-03-10T23:56:19.2510188Z ... 9 more
            dwysakowicz Dawid Wysakowicz added a comment - https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15701&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529&l=41327
            dwysakowicz Dawid Wysakowicz added a comment - This might be related: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15720&view=logs&j=6caf31d6-847a-526e-9624-468e053467d6&t=0b23652f-b18b-5b6e-6eb6-a11070364610&l=29594
            mapohl Matthias Pohl added a comment -

            dwysakowicz: What makes you assume that this failure is related to the ZooKeeper issue of the other build failures reported in this Jira issue?

            mapohl Matthias Pohl added a comment - dwysakowicz : What makes you assume that this failure is related to the ZooKeeper issue of the other build failures reported in this Jira issue?

            Ofc, I might be wrong, but I thought the wall of zookeeper logs looks similar in both cases:

            2021-02-15T01:02:11.3132966Z Feb 15 01:01:58 2021-02-15 00:59:30,586 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3169981Z Feb 15 01:01:58 2021-02-15 00:59:30,586 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3172987Z Feb 15 01:01:58 2021-02-15 00:59:31,732 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3174898Z Feb 15 01:01:58 2021-02-15 00:59:31,732 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3176779Z Feb 15 01:01:58 2021-02-15 00:59:33,661 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00004 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3178828Z Feb 15 01:01:58 2021-02-15 00:59:33,755 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00004 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3180699Z Feb 15 01:01:58 2021-02-15 00:59:37,826 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00003 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3182543Z Feb 15 01:01:58 2021-02-15 00:59:37,826 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00003 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3184497Z Feb 15 01:01:58 2021-02-15 00:59:39,257 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00005 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3186337Z Feb 15 01:01:58 2021-02-15 00:59:39,257 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00005 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3188198Z Feb 15 01:01:58 2021-02-15 00:59:41,054 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00006 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3190043Z Feb 15 01:01:58 2021-02-15 00:59:41,054 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00006 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3191917Z Feb 15 01:01:58 2021-02-15 00:59:43,931 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3214820Z Feb 15 01:01:58 2021-02-15 00:59:43,931 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3216765Z Feb 15 01:01:58 2021-02-15 00:59:45,079 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3218632Z Feb 15 01:01:58 2021-02-15 00:59:45,079 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
            2021-02-15T01:02:11.3220492Z Feb 15 01:01:58 2021-02-15 00:59:48,851 WARN  org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.persistence.FileTxnLog [] - fsync-ing the write ahead log in SyncThread:0 took 2298ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
            
            dwysakowicz Dawid Wysakowicz added a comment - Ofc, I might be wrong, but I thought the wall of zookeeper logs looks similar in both cases: 2021-02-15T01:02:11.3132966Z Feb 15 01:01:58 2021-02-15 00:59:30,586 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3169981Z Feb 15 01:01:58 2021-02-15 00:59:30,586 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3172987Z Feb 15 01:01:58 2021-02-15 00:59:31,732 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3174898Z Feb 15 01:01:58 2021-02-15 00:59:31,732 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3176779Z Feb 15 01:01:58 2021-02-15 00:59:33,661 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00004 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3178828Z Feb 15 01:01:58 2021-02-15 00:59:33,755 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00004 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3180699Z Feb 15 01:01:58 2021-02-15 00:59:37,826 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00003 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3182543Z Feb 15 01:01:58 2021-02-15 00:59:37,826 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00003 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3184497Z Feb 15 01:01:58 2021-02-15 00:59:39,257 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00005 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3186337Z Feb 15 01:01:58 2021-02-15 00:59:39,257 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00005 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3188198Z Feb 15 01:01:58 2021-02-15 00:59:41,054 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00006 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3190043Z Feb 15 01:01:58 2021-02-15 00:59:41,054 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00006 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3191917Z Feb 15 01:01:58 2021-02-15 00:59:43,931 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3214820Z Feb 15 01:01:58 2021-02-15 00:59:43,931 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00000 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3216765Z Feb 15 01:01:58 2021-02-15 00:59:45,079 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - Processing request:: sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3218632Z Feb 15 01:01:58 2021-02-15 00:59:45,079 DEBUG org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.FinalRequestProcessor [] - sessionid:0x10000accbb00001 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a 2021-02-15T01:02:11.3220492Z Feb 15 01:01:58 2021-02-15 00:59:48,851 WARN org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.persistence.FileTxnLog [] - fsync-ing the write ahead log in SyncThread:0 took 2298ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
            mapohl Matthias Pohl added a comment -

            true, thanks for clarification

            mapohl Matthias Pohl added a comment - true, thanks for clarification
            trohrmann Till Rohrmann added a comment -

            Are you looking into this problem mapohl? If yes, then please assign yourself and set the ticket to in progress.

            trohrmann Till Rohrmann added a comment - Are you looking into this problem mapohl ? If yes, then please assign yourself and set the ticket to in progress.
            mapohl Matthias Pohl added a comment -

            The extensive ZooKeeper logs are a result of a wrong log4j setup. I created FLINK-22045 to cover this.

            mapohl Matthias Pohl added a comment - The extensive ZooKeeper logs are a result of a wrong log4j setup. I created FLINK-22045 to cover this.
            mapohl Matthias Pohl added a comment -

            trohrmann you're right. I forgot to assign the issue.

            I looked into the different failures with the help of chesnay. It appears that the job failures are caused by reaching the timeout of the build. This triggers the killing of child processes which results in random failures (e.g. ZooKeeper becoming unavailable). Considering that we're close to the release we might want to increase the timeout rather than putting effort into fixing the build times, I guess?

            mapohl Matthias Pohl added a comment - trohrmann you're right. I forgot to assign the issue. I looked into the different failures with the help of chesnay . It appears that the job failures are caused by reaching the timeout of the build. This triggers the killing of child processes which results in random failures (e.g. ZooKeeper becoming unavailable). Considering that we're close to the release we might want to increase the timeout rather than putting effort into fixing the build times, I guess?
            trohrmann Till Rohrmann added a comment - - edited

            Sounds good to me as a temporary fix. I've assigned the ticket to you now because I accidentally assigned myself yesterday.

            trohrmann Till Rohrmann added a comment - - edited Sounds good to me as a temporary fix. I've assigned the ticket to you now because I accidentally assigned myself yesterday.
            mapohl Matthias Pohl added a comment -

            trohrmann thanks for assigning me.

            dwysakowicz maguowei rmetzger what's your stand on increasing the timeout considering the release? I'm just not sure how we would manage the efforts on improving the runtime of the overall test pipeline afterwards.

            mapohl Matthias Pohl added a comment - trohrmann thanks for assigning me. dwysakowicz maguowei rmetzger what's your stand on increasing the timeout considering the release? I'm just not sure how we would manage the efforts on improving the runtime of the overall test pipeline afterwards.

            Are we sure the problem is with just the test timing out? If I compare execution times from the failed run from description with a last successful build (https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15854&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529) the execution time for the failed test doubles:

            succesful run:

            804;PyFlink YARN per-job on Docker test
            48;Local recovery and sticky scheduling end-to-end test
            129;Local recovery and sticky scheduling end-to-end test
            248;Local recovery and sticky scheduling end-to-end test
            87;Local recovery and sticky scheduling end-to-end test
            358;Local recovery and sticky scheduling end-to-end test
            328;Local recovery and sticky scheduling end-to-end test
            

            failed run:

            848;PyFlink YARN per-job on Docker test
            53;Local recovery and sticky scheduling end-to-end test
            132;Local recovery and sticky scheduling end-to-end test
            601;Local recovery and sticky scheduling end-to-end test <- failed test
            

            BTW by increasing timeout of the build you mean the timeout ~5h timeout for running all e2e tests? There is also something interesting I spotted that I can not explain myself. If you sum up individual execution times of all tests the successful build takes longer. However the Azure dashboard shows that the failed build took ~30 min longer. Do we know why is that?

            dwysakowicz Dawid Wysakowicz added a comment - Are we sure the problem is with just the test timing out? If I compare execution times from the failed run from description with a last successful build ( https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15854&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529 ) the execution time for the failed test doubles: succesful run: 804;PyFlink YARN per-job on Docker test 48;Local recovery and sticky scheduling end-to-end test 129;Local recovery and sticky scheduling end-to-end test 248;Local recovery and sticky scheduling end-to-end test 87;Local recovery and sticky scheduling end-to-end test 358;Local recovery and sticky scheduling end-to-end test 328;Local recovery and sticky scheduling end-to-end test failed run: 848;PyFlink YARN per-job on Docker test 53;Local recovery and sticky scheduling end-to-end test 132;Local recovery and sticky scheduling end-to-end test 601;Local recovery and sticky scheduling end-to-end test <- failed test BTW by increasing timeout of the build you mean the timeout ~5h timeout for running all e2e tests? There is also something interesting I spotted that I can not explain myself. If you sum up individual execution times of all tests the successful build takes longer. However the Azure dashboard shows that the failed build took ~30 min longer. Do we know why is that?
            mapohl Matthias Pohl added a comment -

            You have a point. Comparing it to the successful build is a good idea. ...and I'm back on the whiteboard...

            mapohl Matthias Pohl added a comment - You have a point. Comparing it to the successful build is a good idea. ...and I'm back on the whiteboard...
            mapohl Matthias Pohl added a comment - - edited

            For build #13326 I had a look at the timestamps. First of all, in all cases the test doesn't fail for the HashMapStateBackend. Every failure is caused by the 4 10 rocks false false setup (i.e. parallism=4, maxAttempts=10, EmbeddedRocksDBStateBackend used, incrementalCheckpoints=false, killJvm=false).

            It looks like the checkpoint takes up quite some time for checkpoints later in the test:

            $ grep "Received ack\|attempt #\|Triggering checkpoin" flink-vsts-standalonesession-0-fv-az127-111.log
            2021-02-15 00:52:20,703 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #0) with attempt id 4f4ad74ee089d4dd970ec49c30c04bd9 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:20,720 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #0) with attempt id 82981af733cb7b98a6a5887afe6ddd9f to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:20,725 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #0) with attempt id 449a2c47d0757dcea7dd388e48f85e45 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:20,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #0) with attempt id 7973e2cbeb1c92794d732674a4b61416 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:20,743 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #0) with attempt id 880f6ba3112ff89dc036928542878ce2 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:20,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #0) with attempt id 55bd0597a2f983d09dfc97ff7bd9e825 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:20,767 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #0) with attempt id 82f8ea6e74f36cfd316a288372811f68 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:20,769 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #0) with attempt id 0c95f9805ca3eb643e9dcac59c43f0e7 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:22,930 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 1 (type=CHECKPOINT) @ 1613350342913 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:52:23,900 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 7973e2cbeb1c92794d732674a4b61416 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:23,908 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 449a2c47d0757dcea7dd388e48f85e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:23,949 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 4f4ad74ee089d4dd970ec49c30c04bd9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:52:24,009 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 82981af733cb7b98a6a5887afe6ddd9f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:25,593 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 880f6ba3112ff89dc036928542878ce2 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:52:25,706 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 55bd0597a2f983d09dfc97ff7bd9e825 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:25,734 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 0c95f9805ca3eb643e9dcac59c43f0e7 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:25,859 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 1 from task 82f8ea6e74f36cfd316a288372811f68 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:26,027 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 2 (type=CHECKPOINT) @ 1613350346023 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:52:26,076 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 2 from task 82981af733cb7b98a6a5887afe6ddd9f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:26,109 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 2 from task 7973e2cbeb1c92794d732674a4b61416 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:26,326 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 2 from task 449a2c47d0757dcea7dd388e48f85e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:27,388 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #1) with attempt id 788bb45fc691f093c8fa38aedf4c7c5e to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:27,389 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #1) with attempt id a484233ad0362559effe8505022cd7bb to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:27,389 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #1) with attempt id 405efa0b99caf483b2ff3fbde9626142 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:27,412 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #1) with attempt id ceaba47bd65a109d5ae70453ee6d55d3 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:27,426 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #1) with attempt id 9eef2db0894ded24070d711200cef00b to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:27,430 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #1) with attempt id 11a87877544346f024394efdf94f61a5 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:27,434 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #1) with attempt id bfeb96193c8b4b1a7720523d934ae148 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:27,436 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #1) with attempt id b8789dbe8cb8f86c2a34cbcb1994937e to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:28,312 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 3 (type=CHECKPOINT) @ 1613350348309 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:52:29,130 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task 405efa0b99caf483b2ff3fbde9626142 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:29,230 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task 788bb45fc691f093c8fa38aedf4c7c5e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:52:30,378 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task a484233ad0362559effe8505022cd7bb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:30,969 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task ceaba47bd65a109d5ae70453ee6d55d3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:31,783 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task 11a87877544346f024394efdf94f61a5 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:31,928 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task 9eef2db0894ded24070d711200cef00b of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:52:31,985 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task b8789dbe8cb8f86c2a34cbcb1994937e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:32,078 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 3 from task bfeb96193c8b4b1a7720523d934ae148 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:32,229 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 4 (type=CHECKPOINT) @ 1613350352189 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:52:32,297 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 4 from task ceaba47bd65a109d5ae70453ee6d55d3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:32,304 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 4 from task 405efa0b99caf483b2ff3fbde9626142 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:32,376 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 4 from task a484233ad0362559effe8505022cd7bb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:35,478 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #2) with attempt id 048a1a9d6f2a992a191a13daccb1948c to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:35,479 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #2) with attempt id b0c96b67fcb504a094ad8d0bcbe3e77e to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:35,494 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #2) with attempt id 3b6dfaaa1ca850325860f3862ee79c16 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:35,495 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #2) with attempt id ecfb6e5e16de6b8bc8ebcb4123c01eb0 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:35,502 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #2) with attempt id a98bdd545328575af264df54ec5c26a5 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:35,503 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #2) with attempt id 8b41c40b397ce1c3c4dedf229e317810 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:35,504 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #2) with attempt id efe3eb687b17f610e031d1f30b4703e9 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:35,510 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #2) with attempt id 59fd2b7377356729a937d5a7f1765256 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:36,409 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 5 (type=CHECKPOINT) @ 1613350356406 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:52:41,052 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task 048a1a9d6f2a992a191a13daccb1948c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:52:42,306 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task ecfb6e5e16de6b8bc8ebcb4123c01eb0 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:44,067 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task b0c96b67fcb504a094ad8d0bcbe3e77e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:44,937 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task 3b6dfaaa1ca850325860f3862ee79c16 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:46,971 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task efe3eb687b17f610e031d1f30b4703e9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:47,053 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task 59fd2b7377356729a937d5a7f1765256 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:52:47,081 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task 8b41c40b397ce1c3c4dedf229e317810 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:52:47,359 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 5 from task a98bdd545328575af264df54ec5c26a5 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:52:47,404 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 6 (type=CHECKPOINT) @ 1613350367400 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:52:47,463 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 6 from task 3b6dfaaa1ca850325860f3862ee79c16 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:52:47,551 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 6 from task 048a1a9d6f2a992a191a13daccb1948c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:52:50,769 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #3) with attempt id 42505d412d2fdbfd25b2d2f96185c8cc to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:50,769 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #3) with attempt id 691e4d66555626abcd504ae697a97617 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:50,769 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #3) with attempt id 383562c09b5e1bfd12434ed7e2a17110 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:50,779 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #3) with attempt id 702f75f4ee6a8fbd626272ee4ee69b4d to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:50,779 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #3) with attempt id 016fdf521f8ab31ddbf825098358738b to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:52:50,780 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #3) with attempt id 73add6ac94057d25dd721ceec2edd0fb to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:52:50,783 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #3) with attempt id 02018111834d3bc0ba89bf4a180b6f4e to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:52:50,783 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #3) with attempt id 144e393b9e369541eeb7bed89b79952a to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:52:51,070 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 7 (type=CHECKPOINT) @ 1613350371067 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:53:06,270 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 691e4d66555626abcd504ae697a97617 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:53:06,331 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 42505d412d2fdbfd25b2d2f96185c8cc of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:53:06,959 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 702f75f4ee6a8fbd626272ee4ee69b4d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:53:07,213 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 383562c09b5e1bfd12434ed7e2a17110 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:53:08,936 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 016fdf521f8ab31ddbf825098358738b of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:53:09,125 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 73add6ac94057d25dd721ceec2edd0fb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:53:09,243 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 02018111834d3bc0ba89bf4a180b6f4e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:53:09,602 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 7 from task 144e393b9e369541eeb7bed89b79952a of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:53:09,644 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 8 (type=CHECKPOINT) @ 1613350389632 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:53:14,233 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #4) with attempt id 6f78f2945af7ac21eb1dd3fcf2689ebc to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:53:14,234 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #4) with attempt id f026bd2bbea6227dc80d89b5c6eda695 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:53:14,235 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #4) with attempt id 86a2c8ba4a03b89a2c591cc5d3cd9c6f to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:53:14,235 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #4) with attempt id 58a49f79cc22df7897aed0bbcf83cbf8 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:53:14,235 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #4) with attempt id 78d3ce338fb1e2c27a67fe5eec73fd9a to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:53:14,235 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #4) with attempt id 7900b3067e9211096fa2b9b2ad4db1e3 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:53:14,235 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #4) with attempt id 2beb71300bcfaed2ec16c88312b2bb24 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:53:14,235 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #4) with attempt id 941eb8f47142721812ec335fdac77e66 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:53:15,262 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 9 (type=CHECKPOINT) @ 1613350395260 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:53:32,027 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task f026bd2bbea6227dc80d89b5c6eda695 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:53:32,276 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task 6f78f2945af7ac21eb1dd3fcf2689ebc of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:53:37,030 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task 58a49f79cc22df7897aed0bbcf83cbf8 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:53:42,168 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task 86a2c8ba4a03b89a2c591cc5d3cd9c6f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:53:47,214 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task 941eb8f47142721812ec335fdac77e66 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:53:47,530 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task 2beb71300bcfaed2ec16c88312b2bb24 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:53:47,641 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task 7900b3067e9211096fa2b9b2ad4db1e3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:53:47,944 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 9 from task 78d3ce338fb1e2c27a67fe5eec73fd9a of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:53:48,914 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 10 (type=CHECKPOINT) @ 1613350428898 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:53:48,964 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 10 from task f026bd2bbea6227dc80d89b5c6eda695 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:53:49,135 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 10 from task 6f78f2945af7ac21eb1dd3fcf2689ebc of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #5) with attempt id 910ca61f7469c8ec7949ec14e2c0e239 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #5) with attempt id dfe8fff4ae0da75df98be7d9d6590e45 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #5) with attempt id c50f803c83979afd49ea479f653ba8af to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #5) with attempt id 8c50f0659c3e6541784fdd4c2474333e to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #5) with attempt id 029e69c47b7824deb136eec7728441ef to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #5) with attempt id cb23e8fdefac39a50d3bd5ffb3bc3e86 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #5) with attempt id 95ac0f1a6bf2c2281db6972028d3f957 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:53:51,678 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #5) with attempt id 1d1805b290593d54313a8a7e33d69ab7 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:53:52,142 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 11 (type=CHECKPOINT) @ 1613350432135 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:54:33,436 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task c50f803c83979afd49ea479f653ba8af of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:54:33,540 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task dfe8fff4ae0da75df98be7d9d6590e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:54:34,314 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task 910ca61f7469c8ec7949ec14e2c0e239 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:54:35,783 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task 8c50f0659c3e6541784fdd4c2474333e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:54:41,022 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task 1d1805b290593d54313a8a7e33d69ab7 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:54:41,137 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task cb23e8fdefac39a50d3bd5ffb3bc3e86 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:54:41,179 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task 029e69c47b7824deb136eec7728441ef of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:54:42,511 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 11 from task 95ac0f1a6bf2c2281db6972028d3f957 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:54:43,941 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 12 (type=CHECKPOINT) @ 1613350483847 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:54:43,959 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 12 from task dfe8fff4ae0da75df98be7d9d6590e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:54:44,017 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 12 from task 910ca61f7469c8ec7949ec14e2c0e239 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:54:44,149 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 12 from task c50f803c83979afd49ea479f653ba8af of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:54:48,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #6) with attempt id 8f89a8ab1e618df1b940f7b47dfaf55f to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:54:48,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #6) with attempt id e427bc378c8a4ccef08266f513ba04b3 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:54:48,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #6) with attempt id 3019aa338dcb1c5da07b68f3d2053e8d to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:54:48,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #6) with attempt id fbc19f708a93a3ddcbb4448037af25eb to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:54:48,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #6) with attempt id 1e4fa709d4cae44602d2d0f269f663e6 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:54:48,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #6) with attempt id 5413a5bf4812076e4bcf91a660b2c8df to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:54:48,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #6) with attempt id 7f3cf7692ebb98e99a485fe1c4ca0616 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:54:48,999 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #6) with attempt id c541f3d355a456d9d68456b3393e3109 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:54:49,354 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 13 (type=CHECKPOINT) @ 1613350489351 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:55:34,111 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task 8f89a8ab1e618df1b940f7b47dfaf55f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:55:37,156 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task fbc19f708a93a3ddcbb4448037af25eb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:55:37,666 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task 3019aa338dcb1c5da07b68f3d2053e8d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:55:39,720 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task e427bc378c8a4ccef08266f513ba04b3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:55:45,909 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task c541f3d355a456d9d68456b3393e3109 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:55:46,107 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task 7f3cf7692ebb98e99a485fe1c4ca0616 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:55:46,798 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task 1e4fa709d4cae44602d2d0f269f663e6 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:55:47,181 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 13 from task 5413a5bf4812076e4bcf91a660b2c8df of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:55:49,045 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 14 (type=CHECKPOINT) @ 1613350549043 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:55:49,245 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 14 from task 3019aa338dcb1c5da07b68f3d2053e8d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:55:50,396 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 14 from task 8f89a8ab1e618df1b940f7b47dfaf55f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:55:51,134 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 14 from task fbc19f708a93a3ddcbb4448037af25eb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:55:54,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #7) with attempt id ec5ea040bc73c307ab416a8a1fcb208c to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:55:54,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #7) with attempt id f3f009ae9a8b62967d9cfbe5b3b5f3ad to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:55:54,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #7) with attempt id c67144c6d3b24a2ca98959981dca7507 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:55:54,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #7) with attempt id 9b4d2495d74966513b62f57682ec1b0c to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:55:54,998 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #7) with attempt id 225fc04f7a84880fc83266d12e457d1d to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:55:54,999 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #7) with attempt id a90fc3b1d8f9cf189c769bb82aa9212e to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:55:54,999 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #7) with attempt id 37c71ff6a787d67a389372c90a3ed7cd to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:55:54,999 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #7) with attempt id ed1ee0ebefc90ecc5d49f5bf93cd928b to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:55:55,270 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 15 (type=CHECKPOINT) @ 1613350555266 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:56:47,653 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task ec5ea040bc73c307ab416a8a1fcb208c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:56:50,392 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task 9b4d2495d74966513b62f57682ec1b0c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:56:50,866 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task c67144c6d3b24a2ca98959981dca7507 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:56:52,394 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task f3f009ae9a8b62967d9cfbe5b3b5f3ad of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:57:01,563 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task 37c71ff6a787d67a389372c90a3ed7cd of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:57:01,813 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task ed1ee0ebefc90ecc5d49f5bf93cd928b of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:57:02,145 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task 225fc04f7a84880fc83266d12e457d1d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:57:03,166 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 15 from task a90fc3b1d8f9cf189c769bb82aa9212e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:57:03,840 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 16 (type=CHECKPOINT) @ 1613350623794 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:57:03,897 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 16 from task ec5ea040bc73c307ab416a8a1fcb208c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:57:03,916 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 16 from task 9b4d2495d74966513b62f57682ec1b0c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:57:04,000 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 16 from task c67144c6d3b24a2ca98959981dca7507 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:57:06,478 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 16 from task f3f009ae9a8b62967d9cfbe5b3b5f3ad of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:57:07,038 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #8) with attempt id e9d9b82afa186c411e6bdf881873e25d to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:57:07,038 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #8) with attempt id b35b8153a9f2c0ba60b51ecf243ea73f to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:57:07,038 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #8) with attempt id 39993b609c57b6226c58c199bd722159 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:57:07,038 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #8) with attempt id f979490d52ab8fe2ff77b2932a0bfbef to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:57:07,038 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #8) with attempt id 7e12451b9faa438f6f06e995e474ed71 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:57:07,038 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #8) with attempt id edc0da7ea739f999e8a3c2850d89e5ed to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:57:07,038 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #8) with attempt id 8f96e7dc7d586e7ea101cb16e3facf29 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:57:07,045 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #8) with attempt id 5d7bc3d6c1df759348a120a702e76aa9 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:57:07,423 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 17 (type=CHECKPOINT) @ 1613350627420 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:58:07,870 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task 39993b609c57b6226c58c199bd722159 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:58:07,871 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task b35b8153a9f2c0ba60b51ecf243ea73f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:58:07,886 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task e9d9b82afa186c411e6bdf881873e25d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:58:08,957 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task f979490d52ab8fe2ff77b2932a0bfbef of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:58:19,243 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task 5d7bc3d6c1df759348a120a702e76aa9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:58:19,685 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task 7e12451b9faa438f6f06e995e474ed71 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:58:19,962 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task 8f96e7dc7d586e7ea101cb16e3facf29 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:58:20,276 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 17 from task edc0da7ea739f999e8a3c2850d89e5ed of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:58:21,869 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 18 (type=CHECKPOINT) @ 1613350701857 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:58:21,897 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 18 from task e9d9b82afa186c411e6bdf881873e25d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:58:21,936 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 18 from task f979490d52ab8fe2ff77b2932a0bfbef of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:58:23,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #9) with attempt id 3e8bb0f0ec3899f62304a8dffa103faf to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:58:23,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #9) with attempt id c6864d5e4339977fec5e5b10b8cb9db7 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:58:23,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #9) with attempt id e49c76a882344632c0d9a84d456e2700 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:58:23,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #9) with attempt id 7b95374e5523ec14be73a17668467420 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:58:23,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #9) with attempt id 4bbd2a3aac0de6593f8199b0ef8eaaff to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:58:23,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #9) with attempt id 22514f0ff054d425950b839e92aad3da to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:58:23,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #9) with attempt id adc4277261c074dc802e9f4c5ff80717 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:58:23,363 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #9) with attempt id b3f43cadfe0d2bd2eeff7ba8d2210f10 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:58:23,846 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 19 (type=CHECKPOINT) @ 1613350703843 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:59:26,678 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task 3e8bb0f0ec3899f62304a8dffa103faf of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:59:27,346 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task c6864d5e4339977fec5e5b10b8cb9db7 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:59:35,173 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task 7b95374e5523ec14be73a17668467420 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:59:36,477 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task e49c76a882344632c0d9a84d456e2700 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:59:44,876 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task b3f43cadfe0d2bd2eeff7ba8d2210f10 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 00:59:45,223 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task 22514f0ff054d425950b839e92aad3da of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 00:59:45,824 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task adc4277261c074dc802e9f4c5ff80717 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 00:59:46,549 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 19 from task 4bbd2a3aac0de6593f8199b0ef8eaaff of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 00:59:48,874 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 20 (type=CHECKPOINT) @ 1613350788857 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 00:59:50,794 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #10) with attempt id ad1ef05bb02038bab84960023be2139f to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:59:50,795 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #10) with attempt id 1eb939470ab93f6a50c6df4e0d5757a1 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:59:50,796 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #10) with attempt id 41b6ec89308b04922c9f043faa83c7b9 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:59:50,796 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #10) with attempt id 89878b144856fead9161dafeed1b416d to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:59:50,796 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #10) with attempt id cc83d9b2bb53ac446aad4ed344259223 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 00:59:50,797 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #10) with attempt id 957a638e36686032cb9f9bd7144915f3 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 00:59:50,797 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #10) with attempt id 728e40516877edb97748f12a56b9a4c4 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 00:59:50,809 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #10) with attempt id 072aa4c8afc4326e05d926468125cfac to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 00:59:51,077 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 21 (type=CHECKPOINT) @ 1613350791074 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 01:01:02,644 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task 1eb939470ab93f6a50c6df4e0d5757a1 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 01:01:02,652 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task ad1ef05bb02038bab84960023be2139f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 01:01:11,154 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task 89878b144856fead9161dafeed1b416d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 01:01:12,482 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task 41b6ec89308b04922c9f043faa83c7b9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 01:01:21,515 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task cc83d9b2bb53ac446aad4ed344259223 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 01:01:22,282 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task 728e40516877edb97748f12a56b9a4c4 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525).
            2021-02-15 01:01:23,136 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task 957a638e36686032cb9f9bd7144915f3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557).
            2021-02-15 01:01:23,385 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 21 from task 072aa4c8afc4326e05d926468125cfac of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803).
            2021-02-15 01:01:25,380 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 22 (type=CHECKPOINT) @ 1613350885367 for job dfa837311853a69cdca82db082270dff.
            2021-02-15 01:01:25,631 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Received acknowledge message for checkpoint 22 from task ad1ef05bb02038bab84960023be2139f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283).
            2021-02-15 01:01:28,495 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (1/4) (attempt #11) with attempt id 19a29174acd3e16276da75a060a45f89 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 01:01:28,498 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (2/4) (attempt #11) with attempt id 8ac7c1c87b1809407eec0d64bd20a7cb to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 01:01:28,502 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (3/4) (attempt #11) with attempt id 97f2b0292ba8efb63aea4b44e217f382 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 01:01:28,504 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: Custom Source (4/4) (attempt #11) with attempt id 31d061561664d3b35d4ffb51060783e3 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            2021-02-15 01:01:28,510 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #11) with attempt id a68642cf1f5c0d64625c07ee07dbb57c to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98
            2021-02-15 01:01:28,511 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #11) with attempt id 73854761d80223752d9e973e49899574 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe
            2021-02-15 01:01:28,513 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #11) with attempt id cf6c2d52b4929d79db4a1db09ae7279d to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda
            2021-02-15 01:01:28,514 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #11) with attempt id ebb41ec1aaa5ba8ddf134b3f9ade46ac to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405
            

            sjwiesman do you have a guess here considering your work on FLINK-19467?

            mapohl Matthias Pohl added a comment - - edited For build #13326 I had a look at the timestamps. First of all, in all cases the test doesn't fail for the HashMapStateBackend . Every failure is caused by the 4 10 rocks false false setup (i.e. parallism=4 , maxAttempts=10 , EmbeddedRocksDBStateBackend used, incrementalCheckpoints=false , killJvm=false ). It looks like the checkpoint takes up quite some time for checkpoints later in the test: $ grep "Received ack\|attempt #\|Triggering checkpoin" flink-vsts-standalonesession-0-fv-az127-111.log 2021-02-15 00:52:20,703 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #0) with attempt id 4f4ad74ee089d4dd970ec49c30c04bd9 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:20,720 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #0) with attempt id 82981af733cb7b98a6a5887afe6ddd9f to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:20,725 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #0) with attempt id 449a2c47d0757dcea7dd388e48f85e45 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:20,726 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #0) with attempt id 7973e2cbeb1c92794d732674a4b61416 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:20,743 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #0) with attempt id 880f6ba3112ff89dc036928542878ce2 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:20,763 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #0) with attempt id 55bd0597a2f983d09dfc97ff7bd9e825 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:20,767 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #0) with attempt id 82f8ea6e74f36cfd316a288372811f68 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:20,769 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #0) with attempt id 0c95f9805ca3eb643e9dcac59c43f0e7 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:22,930 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 1 (type=CHECKPOINT) @ 1613350342913 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:52:23,900 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 7973e2cbeb1c92794d732674a4b61416 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:23,908 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 449a2c47d0757dcea7dd388e48f85e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:23,949 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 4f4ad74ee089d4dd970ec49c30c04bd9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:52:24,009 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 82981af733cb7b98a6a5887afe6ddd9f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:25,593 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 880f6ba3112ff89dc036928542878ce2 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:52:25,706 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 55bd0597a2f983d09dfc97ff7bd9e825 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:25,734 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 0c95f9805ca3eb643e9dcac59c43f0e7 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:25,859 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 1 from task 82f8ea6e74f36cfd316a288372811f68 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:26,027 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 2 (type=CHECKPOINT) @ 1613350346023 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:52:26,076 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 2 from task 82981af733cb7b98a6a5887afe6ddd9f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:26,109 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 2 from task 7973e2cbeb1c92794d732674a4b61416 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:26,326 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 2 from task 449a2c47d0757dcea7dd388e48f85e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:27,388 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #1) with attempt id 788bb45fc691f093c8fa38aedf4c7c5e to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:27,389 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #1) with attempt id a484233ad0362559effe8505022cd7bb to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:27,389 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #1) with attempt id 405efa0b99caf483b2ff3fbde9626142 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:27,412 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #1) with attempt id ceaba47bd65a109d5ae70453ee6d55d3 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:27,426 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #1) with attempt id 9eef2db0894ded24070d711200cef00b to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:27,430 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #1) with attempt id 11a87877544346f024394efdf94f61a5 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:27,434 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #1) with attempt id bfeb96193c8b4b1a7720523d934ae148 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:27,436 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #1) with attempt id b8789dbe8cb8f86c2a34cbcb1994937e to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:28,312 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 3 (type=CHECKPOINT) @ 1613350348309 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:52:29,130 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task 405efa0b99caf483b2ff3fbde9626142 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:29,230 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task 788bb45fc691f093c8fa38aedf4c7c5e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:52:30,378 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task a484233ad0362559effe8505022cd7bb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:30,969 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task ceaba47bd65a109d5ae70453ee6d55d3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:31,783 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task 11a87877544346f024394efdf94f61a5 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:31,928 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task 9eef2db0894ded24070d711200cef00b of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:52:31,985 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task b8789dbe8cb8f86c2a34cbcb1994937e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:32,078 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 3 from task bfeb96193c8b4b1a7720523d934ae148 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:32,229 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 4 (type=CHECKPOINT) @ 1613350352189 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:52:32,297 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 4 from task ceaba47bd65a109d5ae70453ee6d55d3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:32,304 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 4 from task 405efa0b99caf483b2ff3fbde9626142 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:32,376 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 4 from task a484233ad0362559effe8505022cd7bb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:35,478 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #2) with attempt id 048a1a9d6f2a992a191a13daccb1948c to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:35,479 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #2) with attempt id b0c96b67fcb504a094ad8d0bcbe3e77e to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:35,494 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #2) with attempt id 3b6dfaaa1ca850325860f3862ee79c16 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:35,495 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #2) with attempt id ecfb6e5e16de6b8bc8ebcb4123c01eb0 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:35,502 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #2) with attempt id a98bdd545328575af264df54ec5c26a5 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:35,503 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #2) with attempt id 8b41c40b397ce1c3c4dedf229e317810 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:35,504 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #2) with attempt id efe3eb687b17f610e031d1f30b4703e9 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:35,510 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #2) with attempt id 59fd2b7377356729a937d5a7f1765256 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:36,409 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 5 (type=CHECKPOINT) @ 1613350356406 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:52:41,052 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task 048a1a9d6f2a992a191a13daccb1948c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:52:42,306 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task ecfb6e5e16de6b8bc8ebcb4123c01eb0 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:44,067 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task b0c96b67fcb504a094ad8d0bcbe3e77e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:44,937 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task 3b6dfaaa1ca850325860f3862ee79c16 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:46,971 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task efe3eb687b17f610e031d1f30b4703e9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:47,053 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task 59fd2b7377356729a937d5a7f1765256 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:52:47,081 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task 8b41c40b397ce1c3c4dedf229e317810 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:52:47,359 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 5 from task a98bdd545328575af264df54ec5c26a5 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:52:47,404 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 6 (type=CHECKPOINT) @ 1613350367400 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:52:47,463 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 6 from task 3b6dfaaa1ca850325860f3862ee79c16 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:52:47,551 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 6 from task 048a1a9d6f2a992a191a13daccb1948c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:52:50,769 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #3) with attempt id 42505d412d2fdbfd25b2d2f96185c8cc to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:50,769 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #3) with attempt id 691e4d66555626abcd504ae697a97617 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:50,769 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #3) with attempt id 383562c09b5e1bfd12434ed7e2a17110 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:50,779 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #3) with attempt id 702f75f4ee6a8fbd626272ee4ee69b4d to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:50,779 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #3) with attempt id 016fdf521f8ab31ddbf825098358738b to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:52:50,780 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #3) with attempt id 73add6ac94057d25dd721ceec2edd0fb to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:52:50,783 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #3) with attempt id 02018111834d3bc0ba89bf4a180b6f4e to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:52:50,783 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #3) with attempt id 144e393b9e369541eeb7bed89b79952a to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:52:51,070 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 7 (type=CHECKPOINT) @ 1613350371067 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:53:06,270 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 691e4d66555626abcd504ae697a97617 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:53:06,331 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 42505d412d2fdbfd25b2d2f96185c8cc of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:53:06,959 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 702f75f4ee6a8fbd626272ee4ee69b4d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:53:07,213 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 383562c09b5e1bfd12434ed7e2a17110 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:53:08,936 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 016fdf521f8ab31ddbf825098358738b of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:53:09,125 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 73add6ac94057d25dd721ceec2edd0fb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:53:09,243 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 02018111834d3bc0ba89bf4a180b6f4e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:53:09,602 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 7 from task 144e393b9e369541eeb7bed89b79952a of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:53:09,644 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 8 (type=CHECKPOINT) @ 1613350389632 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:53:14,233 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #4) with attempt id 6f78f2945af7ac21eb1dd3fcf2689ebc to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:53:14,234 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #4) with attempt id f026bd2bbea6227dc80d89b5c6eda695 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:53:14,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #4) with attempt id 86a2c8ba4a03b89a2c591cc5d3cd9c6f to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:53:14,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #4) with attempt id 58a49f79cc22df7897aed0bbcf83cbf8 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:53:14,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #4) with attempt id 78d3ce338fb1e2c27a67fe5eec73fd9a to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:53:14,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #4) with attempt id 7900b3067e9211096fa2b9b2ad4db1e3 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:53:14,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #4) with attempt id 2beb71300bcfaed2ec16c88312b2bb24 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:53:14,235 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #4) with attempt id 941eb8f47142721812ec335fdac77e66 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:53:15,262 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 9 (type=CHECKPOINT) @ 1613350395260 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:53:32,027 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task f026bd2bbea6227dc80d89b5c6eda695 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:53:32,276 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task 6f78f2945af7ac21eb1dd3fcf2689ebc of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:53:37,030 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task 58a49f79cc22df7897aed0bbcf83cbf8 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:53:42,168 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task 86a2c8ba4a03b89a2c591cc5d3cd9c6f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:53:47,214 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task 941eb8f47142721812ec335fdac77e66 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:53:47,530 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task 2beb71300bcfaed2ec16c88312b2bb24 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:53:47,641 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task 7900b3067e9211096fa2b9b2ad4db1e3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:53:47,944 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 9 from task 78d3ce338fb1e2c27a67fe5eec73fd9a of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:53:48,914 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 10 (type=CHECKPOINT) @ 1613350428898 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:53:48,964 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 10 from task f026bd2bbea6227dc80d89b5c6eda695 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:53:49,135 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 10 from task 6f78f2945af7ac21eb1dd3fcf2689ebc of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #5) with attempt id 910ca61f7469c8ec7949ec14e2c0e239 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #5) with attempt id dfe8fff4ae0da75df98be7d9d6590e45 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #5) with attempt id c50f803c83979afd49ea479f653ba8af to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #5) with attempt id 8c50f0659c3e6541784fdd4c2474333e to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #5) with attempt id 029e69c47b7824deb136eec7728441ef to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #5) with attempt id cb23e8fdefac39a50d3bd5ffb3bc3e86 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #5) with attempt id 95ac0f1a6bf2c2281db6972028d3f957 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:53:51,678 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #5) with attempt id 1d1805b290593d54313a8a7e33d69ab7 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:53:52,142 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 11 (type=CHECKPOINT) @ 1613350432135 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:54:33,436 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task c50f803c83979afd49ea479f653ba8af of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:54:33,540 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task dfe8fff4ae0da75df98be7d9d6590e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:54:34,314 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task 910ca61f7469c8ec7949ec14e2c0e239 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:54:35,783 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task 8c50f0659c3e6541784fdd4c2474333e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:54:41,022 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task 1d1805b290593d54313a8a7e33d69ab7 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:54:41,137 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task cb23e8fdefac39a50d3bd5ffb3bc3e86 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:54:41,179 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task 029e69c47b7824deb136eec7728441ef of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:54:42,511 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 11 from task 95ac0f1a6bf2c2281db6972028d3f957 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:54:43,941 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 12 (type=CHECKPOINT) @ 1613350483847 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:54:43,959 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 12 from task dfe8fff4ae0da75df98be7d9d6590e45 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:54:44,017 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 12 from task 910ca61f7469c8ec7949ec14e2c0e239 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:54:44,149 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 12 from task c50f803c83979afd49ea479f653ba8af of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:54:48,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #6) with attempt id 8f89a8ab1e618df1b940f7b47dfaf55f to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:54:48,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #6) with attempt id e427bc378c8a4ccef08266f513ba04b3 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:54:48,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #6) with attempt id 3019aa338dcb1c5da07b68f3d2053e8d to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:54:48,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #6) with attempt id fbc19f708a93a3ddcbb4448037af25eb to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:54:48,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #6) with attempt id 1e4fa709d4cae44602d2d0f269f663e6 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:54:48,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #6) with attempt id 5413a5bf4812076e4bcf91a660b2c8df to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:54:48,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #6) with attempt id 7f3cf7692ebb98e99a485fe1c4ca0616 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:54:48,999 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #6) with attempt id c541f3d355a456d9d68456b3393e3109 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:54:49,354 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 13 (type=CHECKPOINT) @ 1613350489351 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:55:34,111 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task 8f89a8ab1e618df1b940f7b47dfaf55f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:55:37,156 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task fbc19f708a93a3ddcbb4448037af25eb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:55:37,666 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task 3019aa338dcb1c5da07b68f3d2053e8d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:55:39,720 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task e427bc378c8a4ccef08266f513ba04b3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:55:45,909 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task c541f3d355a456d9d68456b3393e3109 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:55:46,107 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task 7f3cf7692ebb98e99a485fe1c4ca0616 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:55:46,798 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task 1e4fa709d4cae44602d2d0f269f663e6 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:55:47,181 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 13 from task 5413a5bf4812076e4bcf91a660b2c8df of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:55:49,045 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 14 (type=CHECKPOINT) @ 1613350549043 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:55:49,245 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 14 from task 3019aa338dcb1c5da07b68f3d2053e8d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:55:50,396 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 14 from task 8f89a8ab1e618df1b940f7b47dfaf55f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:55:51,134 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 14 from task fbc19f708a93a3ddcbb4448037af25eb of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:55:54,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #7) with attempt id ec5ea040bc73c307ab416a8a1fcb208c to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:55:54,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #7) with attempt id f3f009ae9a8b62967d9cfbe5b3b5f3ad to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:55:54,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #7) with attempt id c67144c6d3b24a2ca98959981dca7507 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:55:54,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #7) with attempt id 9b4d2495d74966513b62f57682ec1b0c to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:55:54,998 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #7) with attempt id 225fc04f7a84880fc83266d12e457d1d to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:55:54,999 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #7) with attempt id a90fc3b1d8f9cf189c769bb82aa9212e to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:55:54,999 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #7) with attempt id 37c71ff6a787d67a389372c90a3ed7cd to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:55:54,999 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #7) with attempt id ed1ee0ebefc90ecc5d49f5bf93cd928b to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:55:55,270 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 15 (type=CHECKPOINT) @ 1613350555266 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:56:47,653 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task ec5ea040bc73c307ab416a8a1fcb208c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:56:50,392 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task 9b4d2495d74966513b62f57682ec1b0c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:56:50,866 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task c67144c6d3b24a2ca98959981dca7507 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:56:52,394 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task f3f009ae9a8b62967d9cfbe5b3b5f3ad of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:57:01,563 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task 37c71ff6a787d67a389372c90a3ed7cd of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:57:01,813 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task ed1ee0ebefc90ecc5d49f5bf93cd928b of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:57:02,145 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task 225fc04f7a84880fc83266d12e457d1d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:57:03,166 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 15 from task a90fc3b1d8f9cf189c769bb82aa9212e of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:57:03,840 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 16 (type=CHECKPOINT) @ 1613350623794 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:57:03,897 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 16 from task ec5ea040bc73c307ab416a8a1fcb208c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:57:03,916 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 16 from task 9b4d2495d74966513b62f57682ec1b0c of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:57:04,000 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 16 from task c67144c6d3b24a2ca98959981dca7507 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:57:06,478 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 16 from task f3f009ae9a8b62967d9cfbe5b3b5f3ad of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:57:07,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #8) with attempt id e9d9b82afa186c411e6bdf881873e25d to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:57:07,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #8) with attempt id b35b8153a9f2c0ba60b51ecf243ea73f to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:57:07,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #8) with attempt id 39993b609c57b6226c58c199bd722159 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:57:07,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #8) with attempt id f979490d52ab8fe2ff77b2932a0bfbef to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:57:07,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #8) with attempt id 7e12451b9faa438f6f06e995e474ed71 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:57:07,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #8) with attempt id edc0da7ea739f999e8a3c2850d89e5ed to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:57:07,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #8) with attempt id 8f96e7dc7d586e7ea101cb16e3facf29 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:57:07,045 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #8) with attempt id 5d7bc3d6c1df759348a120a702e76aa9 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:57:07,423 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 17 (type=CHECKPOINT) @ 1613350627420 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:58:07,870 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task 39993b609c57b6226c58c199bd722159 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:58:07,871 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task b35b8153a9f2c0ba60b51ecf243ea73f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:58:07,886 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task e9d9b82afa186c411e6bdf881873e25d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:58:08,957 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task f979490d52ab8fe2ff77b2932a0bfbef of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:58:19,243 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task 5d7bc3d6c1df759348a120a702e76aa9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:58:19,685 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task 7e12451b9faa438f6f06e995e474ed71 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:58:19,962 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task 8f96e7dc7d586e7ea101cb16e3facf29 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:58:20,276 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 17 from task edc0da7ea739f999e8a3c2850d89e5ed of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:58:21,869 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 18 (type=CHECKPOINT) @ 1613350701857 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:58:21,897 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 18 from task e9d9b82afa186c411e6bdf881873e25d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:58:21,936 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 18 from task f979490d52ab8fe2ff77b2932a0bfbef of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:58:23,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #9) with attempt id 3e8bb0f0ec3899f62304a8dffa103faf to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:58:23,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #9) with attempt id c6864d5e4339977fec5e5b10b8cb9db7 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:58:23,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #9) with attempt id e49c76a882344632c0d9a84d456e2700 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:58:23,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #9) with attempt id 7b95374e5523ec14be73a17668467420 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:58:23,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #9) with attempt id 4bbd2a3aac0de6593f8199b0ef8eaaff to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:58:23,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #9) with attempt id 22514f0ff054d425950b839e92aad3da to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:58:23,333 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #9) with attempt id adc4277261c074dc802e9f4c5ff80717 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:58:23,363 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #9) with attempt id b3f43cadfe0d2bd2eeff7ba8d2210f10 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:58:23,846 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 19 (type=CHECKPOINT) @ 1613350703843 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:59:26,678 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task 3e8bb0f0ec3899f62304a8dffa103faf of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:59:27,346 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task c6864d5e4339977fec5e5b10b8cb9db7 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:59:35,173 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task 7b95374e5523ec14be73a17668467420 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:59:36,477 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task e49c76a882344632c0d9a84d456e2700 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:59:44,876 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task b3f43cadfe0d2bd2eeff7ba8d2210f10 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 00:59:45,223 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task 22514f0ff054d425950b839e92aad3da of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 00:59:45,824 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task adc4277261c074dc802e9f4c5ff80717 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 00:59:46,549 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 19 from task 4bbd2a3aac0de6593f8199b0ef8eaaff of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 00:59:48,874 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 20 (type=CHECKPOINT) @ 1613350788857 for job dfa837311853a69cdca82db082270dff. 2021-02-15 00:59:50,794 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #10) with attempt id ad1ef05bb02038bab84960023be2139f to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:59:50,795 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #10) with attempt id 1eb939470ab93f6a50c6df4e0d5757a1 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:59:50,796 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #10) with attempt id 41b6ec89308b04922c9f043faa83c7b9 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:59:50,796 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #10) with attempt id 89878b144856fead9161dafeed1b416d to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:59:50,796 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #10) with attempt id cc83d9b2bb53ac446aad4ed344259223 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 00:59:50,797 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #10) with attempt id 957a638e36686032cb9f9bd7144915f3 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 00:59:50,797 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #10) with attempt id 728e40516877edb97748f12a56b9a4c4 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 00:59:50,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #10) with attempt id 072aa4c8afc4326e05d926468125cfac to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 00:59:51,077 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 21 (type=CHECKPOINT) @ 1613350791074 for job dfa837311853a69cdca82db082270dff. 2021-02-15 01:01:02,644 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task 1eb939470ab93f6a50c6df4e0d5757a1 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 01:01:02,652 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task ad1ef05bb02038bab84960023be2139f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 01:01:11,154 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task 89878b144856fead9161dafeed1b416d of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 01:01:12,482 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task 41b6ec89308b04922c9f043faa83c7b9 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 01:01:21,515 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task cc83d9b2bb53ac446aad4ed344259223 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 01:01:22,282 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task 728e40516877edb97748f12a56b9a4c4 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525). 2021-02-15 01:01:23,136 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task 957a638e36686032cb9f9bd7144915f3 of job dfa837311853a69cdca82db082270dff at 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557). 2021-02-15 01:01:23,385 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 21 from task 072aa4c8afc4326e05d926468125cfac of job dfa837311853a69cdca82db082270dff at 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803). 2021-02-15 01:01:25,380 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 22 (type=CHECKPOINT) @ 1613350885367 for job dfa837311853a69cdca82db082270dff. 2021-02-15 01:01:25,631 DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Received acknowledge message for checkpoint 22 from task ad1ef05bb02038bab84960023be2139f of job dfa837311853a69cdca82db082270dff at 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283). 2021-02-15 01:01:28,495 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (1/4) (attempt #11) with attempt id 19a29174acd3e16276da75a060a45f89 to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 01:01:28,498 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (2/4) (attempt #11) with attempt id 8ac7c1c87b1809407eec0d64bd20a7cb to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 01:01:28,502 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (3/4) (attempt #11) with attempt id 97f2b0292ba8efb63aea4b44e217f382 to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 01:01:28,504 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: Custom Source (4/4) (attempt #11) with attempt id 31d061561664d3b35d4ffb51060783e3 to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 2021-02-15 01:01:28,510 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (1/4) (attempt #11) with attempt id a68642cf1f5c0d64625c07ee07dbb57c to 10.1.0.4:36079-9ecc1c @ fv-az127-111.internal.cloudapp.net (dataPort=43283) with allocation id 34ba1021819900f400145f4eeb68cf98 2021-02-15 01:01:28,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (2/4) (attempt #11) with attempt id 73854761d80223752d9e973e49899574 to 10.1.0.4:46341-c100f5 @ fv-az127-111.internal.cloudapp.net (dataPort=40557) with allocation id 7dbde52bef472bf82dfe6b9c51030bbe 2021-02-15 01:01:28,513 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (3/4) (attempt #11) with attempt id cf6c2d52b4929d79db4a1db09ae7279d to 10.1.0.4:42443-4be16a @ fv-az127-111.internal.cloudapp.net (dataPort=40525) with allocation id 6bd434b58457e3ede2b354f5ec514bda 2021-02-15 01:01:28,514 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Flat Map -> Sink: Unnamed (4/4) (attempt #11) with attempt id ebb41ec1aaa5ba8ddf134b3f9ade46ac to 10.1.0.4:44619-9da455 @ fv-az127-111.internal.cloudapp.net (dataPort=40803) with allocation id 9e93afdbf0a591fe019a790882e49405 sjwiesman do you have a guess here considering your work on FLINK-19467 ?
            mapohl Matthias Pohl added a comment - - edited

            BTW by increasing timeout of the build you mean the timeout ~5h timeout for running all e2e tests? There is also something interesting I spotted that I can not explain myself. If you sum up individual execution times of all tests the successful build takes longer. However the Azure dashboard shows that the failed build took ~30 min longer. Do we know why is that?

            Yes, I meant increasing the time until the e2e times out. But as you pointed out - there must be another issue with this test.

            About the difference in execution time vs the time which is shown on the Azure dashboard: How did you come up with that? I checked the successful build you shared above:

            $ paste -sd+ <(grep -F "[PASS]" 171-azureci.log | grep -o "passed after [0-9]* minutes and [0-9]*" | cut -d' ' -f 6) | bc
            2597
            $ paste -sd+ <(grep -F "[PASS]" 171-azureci.log | grep -o "passed after [0-9]* minutes and [0-9]*" | cut -d' ' -f 3) | bc
            157
            

            Which means 157mins and 2597secs which boils down to 3h 20mins 17secs in contrast to the AzureCI dashboard providing 3h 36mins 56secs for the overall e2e runtime.

            mapohl Matthias Pohl added a comment - - edited BTW by increasing timeout of the build you mean the timeout ~5h timeout for running all e2e tests? There is also something interesting I spotted that I can not explain myself. If you sum up individual execution times of all tests the successful build takes longer. However the Azure dashboard shows that the failed build took ~30 min longer. Do we know why is that? Yes, I meant increasing the time until the e2e times out. But as you pointed out - there must be another issue with this test. About the difference in execution time vs the time which is shown on the Azure dashboard: How did you come up with that? I checked the successful build you shared above : $ paste -sd+ <(grep -F "[PASS]" 171-azureci.log | grep -o "passed after [0-9]* minutes and [0-9]*" | cut -d ' ' -f 6) | bc 2597 $ paste -sd+ <(grep -F "[PASS]" 171-azureci.log | grep -o "passed after [0-9]* minutes and [0-9]*" | cut -d ' ' -f 3) | bc 157 Which means 157mins and 2597secs which boils down to 3h 20mins 17secs in contrast to the AzureCI dashboard providing 3h 36mins 56secs for the overall e2e runtime.
            sjwiesman Seth Wiesman added a comment -

            I looked at the test and I don't see anything obvious that would point to FLINK-19467. Just to be safe I kicked off a build that reverts the test to use the old RocksDBStateBackend class. If this passes without issue then my change was the problem. You can follow the build here.

             

            https://dev.azure.com/sjwiesman/Flink/_build/results?buildId=56&view=results

            sjwiesman Seth Wiesman added a comment - I looked at the test and I don't see anything obvious that would point to FLINK-19467 . Just to be safe I kicked off a build that reverts the test to use the old RocksDBStateBackend class. If this passes without issue then my change was the problem. You can follow the build here.   https://dev.azure.com/sjwiesman/Flink/_build/results?buildId=56&view=results
            mapohl Matthias Pohl added a comment -

            I looked at the test and I don't see anything obvious that would point to FLINK-19467. Just to be safe I kicked off a build that reverts the test to use the old RocksDBStateBackend class. If this passes without issue then my change was the problem. You can follow the build here.

            I guess that's not what we can conclude, unfortunately, since it's not constantly failing. pnowojski could someone of you have a look at this?

            mapohl Matthias Pohl added a comment - I looked at the test and I don't see anything obvious that would point to FLINK-19467 . Just to be safe I kicked off a build that reverts the test to use the old RocksDBStateBackend class. If this passes without issue then my change was the problem. You can follow the build here. I guess that's not what we can conclude, unfortunately, since it's not constantly failing. pnowojski could someone of you have a look at this?
            mapohl Matthias Pohl added a comment - - edited

            I unassigned myself after verifying with AHeise. It's in the runtime team's backlog.

            mapohl Matthias Pohl added a comment - - edited I unassigned myself after verifying with AHeise . It's in the runtime team's backlog.

            From what I see in the logs:
            1. Test times out because the needed 10 checkpoints take too much time each (seconds in PASS case vs tens of seconds to minutes in FAIL case)
            2. Checkpoints take longer either because of either longer alignment (~20s) or  longer time to trigger checkpoint at the source (40-55s). Both indicate high back-pressure:

            2021-02-08 22:27:13,093 DEBUG org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl [] - Flat Map -> Sink: Unnamed (1/4)#3 - finished synchronous part of checkpoint 7. Alignment duration: 21525 ms, snapshot duration -1 ms, is unaligned checkpoint : false
            
            2021-02-08 22:27:19,560 DEBUG org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Trigger checkpoint 9@1612823239549 for 4343c6f1810b6f077dd688312b88e60c.
            2021-02-08 22:28:06,724 DEBUG org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl [] - Source: Custom Source (2/4)#4 - finished synchronous part of checkpoint 9. Alignment duration: 0 ms, snapshot duration -1 ms, is unaligned checkpoint : false
            

            Nor sync neither async phases take more than 5s.

             

            Sources generate data without any delay currently, so back-pressure can happen very likely if the machine is slow. The test tries to perform 10 checkpoints in 10 minutes; if each takes a bit more than 1 minute then it's likely to timeout. This can also explain why RocksDB fails and HashTable doesn't (the latter works faster).

             

            So propose:
            1. Increase test timeout from 10m to 15m
            2. Induce delay between emitting elements in sources (currently none,  change to 100ms)
            3. Log time from receiving checkpoint RPC to actually executing it if it exceeds some threshold (say 10s, WARN level)

            Turning on UC could eliminate long alignment but not the delay of triggering checkpoint at the sources. So wouldn't enable it and wait for it to become a default.
             

            roman Roman Khachatryan added a comment - From what I see in the logs: 1. Test times out because the needed 10 checkpoints take too much time each (seconds in PASS case vs tens of seconds to minutes in FAIL case) 2. Checkpoints take longer either because of either longer alignment (~20s) or  longer time to trigger checkpoint at the source (40-55s). Both indicate high back-pressure: 2021-02-08 22:27:13,093 DEBUG org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl [] - Flat Map -> Sink: Unnamed (1/4)#3 - finished synchronous part of checkpoint 7. Alignment duration: 21525 ms, snapshot duration -1 ms, is unaligned checkpoint : false 2021-02-08 22:27:19,560 DEBUG org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Trigger checkpoint 9@1612823239549 for 4343c6f1810b6f077dd688312b88e60c. 2021-02-08 22:28:06,724 DEBUG org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl [] - Source: Custom Source (2/4)#4 - finished synchronous part of checkpoint 9. Alignment duration: 0 ms, snapshot duration -1 ms, is unaligned checkpoint : false Nor sync neither async phases take more than 5s.   Sources generate data without any delay currently, so back-pressure can happen very likely if the machine is slow. The test tries to perform 10 checkpoints in 10 minutes; if each takes a bit more than 1 minute then it's likely to timeout. This can also explain why RocksDB fails and HashTable doesn't (the latter works faster).   So propose: 1. Increase test timeout from 10m to 15m 2. Induce delay between emitting elements in sources (currently none,  change to 100ms) 3. Log time from receiving checkpoint RPC to actually executing it if it exceeds some threshold (say 10s, WARN level) Turning on UC could eliminate long alignment but not the delay of triggering checkpoint at the sources. So wouldn't enable it and wait for it to become a default.  
            roman Roman Khachatryan added a comment - Merged into master as https://github.com/apache/flink/commit/2f4d9d8d3cf781ee03e60cc21db0addb9ebcc0f4
            maguowei Guowei Ma added a comment - https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18107&view=logs&j=08866332-78f7-59e4-4f7e-49a56faa3179&t=7f606211-1454-543c-70ab-c7a028a1ce8c&l=37432 Maybe we could merge the fix to the 1.12?
            roman Roman Khachatryan added a comment - - edited

            Also merged into 1.12 as e09c919103a19b97abd169732e44ef7231fab1be

            and into 1.13 as 2f1e7928258876970ac9859e7afe070c9f03fa96

            roman Roman Khachatryan added a comment - - edited Also merged into 1.12 as e09c919103a19b97abd169732e44ef7231fab1be and into 1.13 as 2f1e7928258876970ac9859e7afe070c9f03fa96

            People

              roman Roman Khachatryan
              rmetzger Robert Metzger
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: