Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-23548

Flink Session Cluster (Standalone) in Kubernetes: Exception occurred in REST handler: Could not execute application.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: 1.13.0, 1.13.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Release Note:
      Resolved by attaching PVC volume and outputting logs there in detached mode.

      Description

      Following the guide located here: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/ to deploy Session Cluster in standalone mode on Kubernetes seems to produce weird REST errors in the UI. When running any of the examples through the UI and submitting jobs the Job Manager will show: Exception occurred in REST handler: Could not execute application while the UI produces the bellow stack trace.

      I will mention the the job completes successfully and submitting the job through command line also works correctly but we are attempting to figure out why the UI/REST errors are appearing and if there is a possible fix in the kubernetes yaml definitions that are provided in the guide above. This is painful for us due to many of our users submitting jobs through the UI.

       

      Job Manager Pod Error:

      2021-07-29 21:03:01,982 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Reduce (SUM(1), at main(WordCount.java:87) (1/1) (attempt #0) with attempt id e626567e699fcfafd5902 
      2021-07-29 21:03:01,983 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - DataSink (collect()) (1/1) (2461dbbb123ea836d88d3bba293c5b2c) switched from SCHEDULED to DEPLOYING.           
      2021-07-29 21:03:01,985 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying DataSink (collect()) (1/1) (attempt #0) with attempt id 2461dbbb123ea836d88d3bba293c5b2c to 10.30.4 
      2021-07-29 21:03:02,002 WARN  org.apache.flink.client.deployment.application.DetachedApplicationRunner [] - Could not execute application:                                                                    
      org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Job was submitted in detached mode. Results of job execution, such as accumulators, runtime, etc. are not availa 
          at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                                          
          at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                       
          at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                                                          
          at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                        
          at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                   
          at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                         
          at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) [?:1.8.0_292]                                                                                                      
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]                                                                                                                  
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]                                                                                                                                 
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]                                                                    
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_292]                                                                           
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]                                                                                                          
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]                                                                                                          
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]                                                                                                                                                    
      Caused by: org.apache.flink.api.common.InvalidProgramException: Job was submitted in detached mode. Results of job execution, such as accumulators, runtime, etc. are not available. Please make sure your pr 
          at org.apache.flink.core.execution.DetachedJobExecutionResult.getAccumulatorResult(DetachedJobExecutionResult.java:56) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                               
          at org.apache.flink.api.java.DataSet.collect(DataSet.java:419) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                                                                       
          at org.apache.flink.api.java.DataSet.print(DataSet.java:1748) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                                                                        
          at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:96) ~[?:?]                                                                                                                      
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]                                                                                                                             
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]                                                                                                           
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]                                                                                                   
          at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]                                                                                                                                        
          at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist_2.11-1.13.0.jar:1.13.0]                                                                          
          ... 13 more                                                                                                                                                                                               
      2021-07-29 21:03:02,004 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler   [] - Exception occurred in REST handler: Could not execute application.
      

       

      UI Stack Error:

      org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:108) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ... 7 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Job was submitted in detached mode. Results of job execution, such as accumulators, runtime, etc. are not available. Please make sure your program doesn't call an eager execution function [collect, print, printToErr, count]. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ... 10 more Caused by: org.apache.flink.api.common.InvalidProgramException: Job was submitted in detached mode. Results of job execution, such as accumulators, runtime, etc. are not available. Please make sure your program doesn't call an eager execution function [collect, print, printToErr, count]. at org.apache.flink.core.execution.DetachedJobExecutionResult.getAccumulatorResult(DetachedJobExecutionResult.java:56) at org.apache.flink.api.java.DataSet.collect(DataSet.java:419) at org.apache.flink.api.java.DataSet.print(DataSet.java:1748) at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ... 13 more
      

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zx10r zx10r
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: