[FLINK-31135] ConfigMap DataSize went > 1 MB and cluster stopped working

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: kubernetes-operator-1.2.0
• Component/s: Kubernetes Operator
• Assignee: Unassigned
• Reporter: Sriram Ganesh
• Created: 20/Feb/23 10:12
• Updated: 04/May/23 13:39
• Resolved: 04/May/23 13:39

Description

I am Flink Operator to manage clusters. Flink version: 1.15.2. Flink jobs failed with the below error. It seems the config map size went beyond 1 MB (default size). 

Since it is managed by the operator and config maps are not updated with any manual intervention, I suspect it could be an operator issue. 

 

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://<IP>/api/v1/namespaces/<NS>/configmaps/<job>-config-map. Message: ConfigMap "<job>-config-map" is invalid: []: Too long: must have at most 1048576 bytes. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=[], message=Too long: must have at most 1048576 bytes, reason=FieldValueTooLong, additionalProperties={})], group=null, kind=ConfigMap, name=<job>-config-map, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=ConfigMap "<job>-config-map" is invalid: []: Too long: must have at most 1048576 bytes, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:673) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:560) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:347) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:327) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:781) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:183) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:188) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:130) ~[flink-dist-1.15.2.jar:1.15.2]
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:41) ~[flink-dist-1.15.2.jar:1.15.2]
at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$attemptCheckAndUpdateConfigMap$11(Fabric8FlinkKubeClient.java:325) ~[flink-dist-1.15.2.jar:1.15.2]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
... 3 more