[FLINK-12160] Support to ship user config files on kubernetes

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Runtime / Coordination
• Assignee: Yang Wang
• Reporter: Yang Wang
• Created: 11/Apr/19 01:40
• Updated: 08/Apr/20 03:28
• Resolved: 08/Apr/20 03:28

Description

When deploying flink cluster on Yarn, `yarnShip` cli option could be used to ship user config files and jars. The files are registered as Yarn local resource which is saved on hdfs.

 

However, we do not have a default existing dfs. The `ConfigMap` could be used to ship small files. The files will be stored in etcd and then mounted to jobmanager and taskmanager pod so that they could use it as locally.

-kt,--kubernetesship <arg>                 Ship files in the specified directory
                                          (t for transfer). Only small files(<1MB) could be supported.

https://stackoverflow.com/questions/53012798/kubernetes-configmap-size-limitation