[FLINK-9612] Add option for minimal artifacts being pulled in Mesos

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Won't Do
• Component/s: Deployment / Mesos, flink-docker, Runtime / Configuration
• Labels: auto-deprioritized-major
• Assignee: Unassigned
• Reporter: Addison Higham
• Created: 19/Jun/18 01:44
• Updated: 30/Apr/21 09:26
• Resolved: 30/Apr/21 09:26

Description

NOTE: this assumes mesos, but this improvement could also be useful for future container deployments.

Currently, in mesos, the FlinkDistributionOverlay  copies the entire `conf`, `bin`, and `lib` folders from the running JobManager/ResourceManager. When using docker with a pre-installed flink distribution, this is relatively inefficient as it pulls jars that are already baked into the container image.

A new option that disables pulling most (if not all?) of the FlinkDistributionOverlay could allow for much faster and more scalable provisions of TaskManagers. As it currently stands, trying to run a few hundred TaskManagers is likely to result in poor performance in pulling all the artifacts from the MesosArtifactServer