Shane Kumpf, Varun Vasudev, Zhankun Tang,
I read through the attached design doc. My thoughts follow.
We already have an existing model on how to deal with artifact localization:
A client (end-user clients or ApplicationMasters) ask NodeManagers to launch containers
As part of launching containers, they specify two things in order for the NodeManager to download the requisite artifacts (java API LocalResource):
- The list of artifacts - files, jars, tar-balls etc each qualified as a URI
- The URI dictates to the NodeManager where the artifact is and the schema informs how to get to that path
- NodeManager today only understands Hadoop FileSystem specific URIs.
- In addition to the URI, a bunch of credentials that the user himself/herself passes along to the NodeManager saying "here, use these credentials to talk to the remote storage, representing me, and download the files on the local box. By the way, keep the credentials safe."
Why can't we extend this mechanism from the API perspective? Kind of your approach (4.2) in the doc.
- Extend the URL notion to include docker paths
- Pass the login credentials also along similar to our Container credentials today. Obviously, this needs more plumbing to be able to pass along docker login credentials.
- Other non-secretive parameters can be passed along as another configuration file in distributed-cache. This obviously creates another need for yet-another storage besides docker, but we already need this to be able to kickstart the AM, spread around more config files etc.
Your argument about every framework needing to do some work is valid. But almost all of today's frameworks already have user-defined configuration properties to specify additional distributed-cache files without changing code everytime.
And finally, today, NodeManager doesn't have a notion of a default artifacts store. Every app specifies where its files are. We should try to keep this. If not, we should add a first class notion of default artifacts store both for docker and non-docker containers.