Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33926

Can't start a job with a jar in the system classpath in native k8s mode

    XMLWordPrintableJSON

Details

    Description

      It appears that the combination of the running operator-controlled jobs in native k8s + application mode + using a job jar in the classpath is invalid. Avoiding dynamic classloading (as specified in the docs) is beneficial for some jobs. This affects at least Flink 1.16.1 and Kubernetes Operator 1.6.0.

       

      FLINK-29288 seems to have addressed this for standalone mode. If I am misunderstanding how to correctly build jars for this native k8s scenario, apologies for the noise and any pointers would be appreciated!

       

      Perhaps related, the spec documentation declares it optional, but isn't clear about under what conditions that applies.

      • Putting the jar in the system classpath and pointing jarURI to that jar leads to linkage errors.
      • Not including jarURI leads to NullPointerExceptions in the operator:
      {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException: java.lang.NullPointerException\n\tat org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: java.lang.NullPointerException\n\tat org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}  

      Attachments

        Issue Links

          Activity

            People

              tinaselenge Gantigmaa Selenge
              trystan Trystan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: