[SUBMARINE-148] [Umbrella] Submarine-sdk tracking

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: SDK
• Assignee: Kevin Su
• Reporter: Kevin Su
• Created: 26/Aug/19 07:35
• Updated: 22/Sep/20 04:16
• Resolved: 22/Sep/20 04:16

Description

The submarine-sdk support Python, Scala, R language for algorithm development, The SDK is provided to help developers use submarine's internal data caching, data exchange, and task tracking to more efficiently improve the development and execution of machine learning tasks.

 

pysubmarine tracking module is a python API for logging parameters, code versions, metrics

1. Add the sumbrine-sdk/pysubmarine module.
2. pysubmarine supports mysql/HDFS in two ways to store metric data. Priority to do mysql, in order to performance requires python to directly write mysql data, do not take the submarine-sever RESTful interface.
3. Add metric, param instance to the submarine-server database and rest
4. Add Metric tracking ui to submarine-web
5. Add a metric graphical display to submarine-web

Issue Links

• is related to: SUBMARINE-98 [Umbrella] Submarine web page
• is related to: SUBMARINE-165 [Umbrella] Submarine Workbench
• links to: PySubmarine design doc

Sub-Tasks

• [SDK] Add submarine-sdk module: Resolved
• [SDK] Support mysql as the storage backend: Resolved
• [SDK] Support hdfs as the storage backend: Resolved
• [SDK] Update pysubmarine-tracking doc: Resolved
• [SDK] Add pysubmarine-tracking example in mini-submarine: Resolved
• [SDK] support python 3.5, 3.6, 3.7, 3.8: Resolved
• Add metric, param instance to the submarine-server rest: Resolved
• [SDK] Set job_name to id in database: Resolved