[SUBMARINE-857] [Umbrella] Support model management SDK in distributed scenerios

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Labels: pull-request-available
• Target Version: 0.6.0
• Assignee: Byron Hsu
• Reporter: Byron Hsu
• Created: 21/Jun/21 09:52
• Updated: 02/Jul/23 08:49
• Resolved: 29/Jun/21 09:17

Description

Submarine is a platform designed for distributed training, so its model management SDK should be easier to use in distributed scenarios.

In a general distributed experiment, there are several workers training together.

Our model management toolkit will support:
1. The workers in the same experiment will automatically direct their logs to the same group in mlflow, so users can monitor multiple workers' info in one graph.
2. When saving models, users do not need to store all the workers' because some are replicated or redundant. Calling save_model in our toolkit, we will apply the most efficient saving strategy under the hood, which can cost the least space and time.

The API design doc can be viewed here: https://hackmd.io/I6frSeZIQDaKQYK4nGCR5w?both

Issue Links

• links to: GitHub Pull Request #613

Sub-Tasks

• Implement log_param and log_metric for distributed scenarios: Resolved
• Mlflow sqlite database should be replaced with mysql: Resolved
• Replace hard-coded IP address with DNS in Mlflow and minio service: Resolved
• Trim the size of pysubmarine: Resolved
• Support log_params and log_metrics: Resolved
• Clean up deleted experiment in mlflow: Resolved
• Add test mlflow mysql backend test: Resolved
• Image should be built locally in github-action.: Resolved
• Add save model to pysubmarine ModelsClient: Resolved
• Add distributive pytorch example: Resolved
• A time lag between the frontend and backend of MLflow: Resolved
• ServeModel API in submarine server: Resolved
• Tensorflow distributed training example with model management API: Resolved
• Remove hard-coded namespace from ServeRequest Spec: Resolved
• Add handling of non-existing model in model serve.: Open
• Add API to get the serve pod's status.: Open