[SUBMARINE-857] [Umbrella] Support model management SDK in distributed scenerios - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.6.0
Component/s: None
Labels:
- pull-request-available

Target Version:

0.6.0

Description

Submarine is a platform designed for distributed training, so its model management SDK should be easier to use in distributed scenarios.

In a general distributed experiment, there are several workers training together.

Our model management toolkit will support:
1. The workers in the same experiment will automatically direct their logs to the same group in mlflow, so users can monitor multiple workers' info in one graph.
2. When saving models, users do not need to store all the workers' because some are replicated or redundant. Calling save_model in our toolkit, we will apply the most efficient saving strategy under the hood, which can cost the least space and time.

The API design doc can be viewed here: https://hackmd.io/I6frSeZIQDaKQYK4nGCR5w?both

Attachments

Issue Links

links to

GitHub Pull Request #613

Sub-Tasks

1.	Implement log_param and log_metric for distributed scenarios	Resolved	Byron Hsu
2.	Mlflow sqlite database should be replaced with mysql	Resolved	Jhan-Shuo Liu
3.	Replace hard-coded IP address with DNS in Mlflow and minio service	Resolved	Jhan-Shuo Liu
4.	Trim the size of pysubmarine	Resolved	Liang-Chun, Chen
5.	Support log_params and log_metrics	Resolved	Yuan-Chia, Chang
6.	Clean up deleted experiment in mlflow	Resolved	Jhan-Shuo Liu
7.	Add test mlflow mysql backend test	Resolved	Unassigned
8.	Image should be built locally in github-action.	Resolved	Byron Hsu
9.	Add save model to pysubmarine ModelsClient	Resolved	Jhan-Shuo Liu
10.	Add distributive pytorch example	Resolved	Liang-Chun, Chen
11.	A time lag between the frontend and backend of MLflow	Resolved	You Ting Wang
12.	ServeModel API in submarine server	Resolved	Byron Hsu
13.	Tensorflow distributed training example with model management API	Resolved	Liang-Chun, Chen
14.	Remove hard-coded namespace from ServeRequest Spec	Resolved	Jhan-Shuo Liu
15.	Add handling of non-existing model in model serve.	Open	Jhan-Shuo Liu
16.	Add API to get the serve pod's status.	Open	Jhan-Shuo Liu

Activity

People

Assignee:: Byron Hsu

Reporter:: Byron Hsu

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 21/Jun/21 09:52

Updated:: 02/Jul/23 08:49

Resolved:: 29/Jun/21 09:17