Major Description
Currently the flink write task parallelism is set up throughÂ
write.tasks
it is kind of a fixed number during the lifecycle of the ingestion pipeline, while for streaming, there are always fluctuation of the workload, it is great if we can tune the parallelism of write tasks based on the job load profile dynamically.
On K8s, Flink provides a https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ which is suitable for the purpose, which deserves a further investigation.