[SUBMARINE-570] Support run experiment/notebook with synced code

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: code-sync
• Labels: pull-request-available
• Target Version: 0.5.0
• Assignee: Manikandan R
• Reporter: Wangda Tan
• Created: 19/Jul/20 18:07
• Updated: 18/Aug/20 01:49
• Resolved: 18/Aug/20 01:49

Description

In Submarine Experiment, there're needs to sync up code from Git/S3 to run experiment. 

For example, Experiment can be defined to: 

experiment: #Or notebook, model-serving
       name: "abc",
       environment: "team-default-ml-env"
       // Other fields
       code:
           sync_mode: git
           url: "https://foo.com/training-job.git"

And once the experiment is launched, code from https://foo.com/training-job.git will be synced under /code directory of the launched container. 

Different sync_mode can be supported, it can be git, s3, hdfs, nfs, etc.

Similarily, notebook can sync code when a notebook session is launched, this avoids user manual checkout code post notebook session launch. 

More design can be found at the following design docs:

• https://github.com/apache/submarine/blob/master/docs/design/storage-implementation.md#store-code-for-experimentnotebookmodel-serving

- https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#api-of-experiment

Issue Links

• links to: GitHub Pull Request #362
• mentioned in: Page Loading...

Sub-Tasks

• Support for "ssh" based git sync mode: Resolved
• Support for git private repositories: Open