[FLINK-9611] Allow for user-defined artifacts to be specified as part of a mesos overlay

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Abandoned
• Affects Version/s: 1.5.0
• Component/s: Deployment / Mesos, flink-docker, Runtime / Configuration
• Assignee: Unassigned
• Reporter: Addison Higham
• Created: 19/Jun/18 01:37
• Updated: 06/Nov/20 10:31
• Resolved: 06/Nov/20 10:31

Description

NOTE: this assumes mesos, but this improvement could also be useful for future container deployments.

Currently, when deploying to mesos, the "Overlay" functionality is used to determine which artifacts are to be downloaded into the container. However, there isn't a way to plug in your own artifacts to be downloaded into the container. This can cause problems with certain deployment models. 

For example, if you are running flink in docker on mesos, you cannot easily use a private docker image. Typically with mesos and private docker images, you specify credentials as a URI to be downloaded into the container that give permissions to download the private image. Typically, this credentials expire after a few days, so baking them into a docker host isn't a solution.

It would make sense to add a `MesosUserOverlay` that would simplify take some new configuration parameters and add any custom artifacts (or possibly also environment variables?) 

Another solution (or longer term solution) might be to allow for dynamically loading an overlay class for even further customization of the container specification.