Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34561

Downgrading flink-kubernetes-operator causes failure

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • None
    • None
    • Kubernetes Operator
    • None

    Description

      Hi,

      We are currently using flink-kubernetes-operator 1.4 version (with v1beta1 CRD) to manage flink deployments.

      We tried an upgrade to version 1.6 version, but when we try to rollback to 1.4, noticed that flink-operator fails to come up with this execption:

      Exception in thread "main" io.javaoperatorsdk.operator.OperatorException: Error starting operator
      	at io.javaoperatorsdk.operator.Operator.start(Operator.java:125)
      	at org.apache.flink.kubernetes.operator.FlinkOperator.run(FlinkOperator.java:215)
      	at org.apache.flink.kubernetes.operator.FlinkOperator.main(FlinkOperator.java:229)
      Caused by: io.javaoperatorsdk.operator.OperatorException: java.lang.IllegalStateException: com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize spec, this indicates a bug... (through reference chain: org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
      	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$2(ExecutorServiceManager.java:107)
      	at java.base/java.util.ArrayList.forEach(Unknown Source)
      	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.executeAndWaitForAllToComplete(ExecutorServiceManager.java:102)
      	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.boundedExecuteAndWaitForAllToComplete(ExecutorServiceManager.java:82)
      	at io.javaoperatorsdk.operator.ControllerManager.start(ControllerManager.java:36)
      	at io.javaoperatorsdk.operator.Operator.start(Operator.java:120)
      	... 2 more
      Caused by: java.lang.IllegalStateException: com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize spec, this indicates a bug... (through reference chain: org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
      	at io.javaoperatorsdk.operator.api.config.ConfigurationService$1.clone(ConfigurationService.java:112)
      	at java.base/java.util.Optional.map(Unknown Source)
      	at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.get(InformerManager.java:175)
      	at io.javaoperatorsdk.operator.processing.event.source.informer.ManagedInformerEventSource.get(ManagedInformerEventSource.java:110)
      	at io.javaoperatorsdk.operator.processing.event.EventProcessor.submitReconciliationExecution(EventProcessor.java:128)
      	at io.javaoperatorsdk.operator.processing.event.EventProcessor.handleMarkedEventForResource(EventProcessor.java:120)
      	at io.javaoperatorsdk.operator.processing.event.EventProcessor.handleAlreadyMarkedEvents(EventProcessor.java:376)
      	at io.javaoperatorsdk.operator.processing.event.EventProcessor.start(EventProcessor.java:371)
      	at io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:336)
      	at io.javaoperatorsdk.operator.ControllerManager.lambda$start$0(ControllerManager.java:37)
      	at io.javaoperatorsdk.operator.api.config.ExecutorServiceManager.lambda$executeAndWaitForAllToComplete$0(ExecutorServiceManager.java:96)
      	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.base/java.lang.Thread.run(Unknown Source)
      Caused by: com.fasterxml.jackson.databind.JsonMappingException: Could not deserialize spec, this indicates a bug... (through reference chain: org.apache.flink.kubernetes.operator.api.FlinkDeployment["status"]->org.apache.flink.kubernetes.operator.api.status.FlinkDeploymentStatus["lifecycleState"])
      	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402)
      	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361)
      	at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
      	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:782)
      	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
      	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:733)
      	at io.fabric8.kubernetes.model.jackson.BeanPropertyWriterDelegate.serializeAsField(BeanPropertyWriterDelegate.java:66)
      	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
      	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
      	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
      	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
      	at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4624)
      	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3869)
      	at io.javaoperatorsdk.operator.api.config.ConfigurationService$1.clone(ConfigurationService.java:110)
      	... 14 more
      Caused by: java.lang.RuntimeException: Could not deserialize spec, this indicates a bug...
      	at org.apache.flink.kubernetes.operator.api.utils.SpecUtils.deserializeSpecWithMeta(SpecUtils.java:64)
      	at org.apache.flink.kubernetes.operator.api.status.ReconciliationStatus.deserializeLastReconciledSpecWithMeta(ReconciliationStatus.java:71)
      	at org.apache.flink.kubernetes.operator.api.status.ReconciliationStatus.deserializeLastReconciledSpec(ReconciliationStatus.java:59)
      	at org.apache.flink.kubernetes.operator.api.status.CommonStatus.getLifecycleState(CommonStatus.java:75)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:689)
      	at io.fabric8.kubernetes.model.jackson.BeanPropertyWriterDelegate.serializeAsField(BeanPropertyWriterDelegate.java:66)
      	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:774)
      	... 24 more
      Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "ephemeralStorage" (class org.apache.flink.kubernetes.operator.api.spec.Resource), not marked as ignorable (2 known properties: "cpu", "memory"])
       at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.apache.flink.kubernetes.operator.api.spec.FlinkDeploymentSpec["jobManager"]->org.apache.flink.kubernetes.operator.api.spec.JobManagerSpec["resource"]->org.apache.flink.kubernetes.operator.api.spec.Resource["ephemeralStorage"])
      	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
      	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1132)
      	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2202)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1705)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1683)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
      	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
      	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
      	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
      	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4706)
      	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2879)
      	at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3343)
      	at org.apache.flink.kubernetes.operator.api.utils.SpecUtils.deserializeSpecWithMeta(SpecUtils.java:60)
      	... 34 more
      

      Looks like some FlinkDeployments were created by 1.6 version and later 1.4 was able to handle those.

      But as perĀ docs, the FlinkDeployment created by 1.6 should have been backward compatible with 1.4.

      Starting from v1beta1 (operator version 1.0.0) we aim to provide backward compatibility for the already deployed Flink custom resources (FlinkDeployment, FlinkSessionJob).

      Attachments

        Activity

          People

            Unassigned Unassigned
            asanwal Avi Sanwal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: