[SUBMARINE-2] Hadoop {Submarine} Project: Simple and scalable deployment of deep learning training / serving jobs on Hadoop

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 09/Apr/18 21:37
• Updated: 17/Dec/19 04:04
• Resolved: 17/Dec/19 04:04

Description

Description:

Goals:

• Allow infra engineer / data scientist to run unmodified Tensorflow jobs on YARN.
• Allow jobs easy access data/models in HDFS and other storages.
• Can launch services to serve Tensorflow/MXNet models.
• Support run distributed Tensorflow jobs with simple configs.
• Support run user-specified Docker images.
• Support specify GPU and other resources.
• Support launch tensorboard if user specified.
• Support customized DNS name for roles (like tensorboard.$user.$domain:6006)

Why this name?

• Because Submarine is the only vehicle can let human to explore deep places. B-)

Please refer to on-going design doc, and add your thoughts: https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#

 

Submarine all document list in here: https://issues.apache.org/jira/browse/SUBMARINE-113 

 

See Also:

• Zeppelin integration with Submarine design: https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#heading=h.4jov859x47qe

Issue Links

• duplicates: YARN-8220 Running Tensorflow on YARN with GPU and Docker - Examples
• duplicates: YARN-8237 mxnet yarn spec file to add to native service examples
• duplicates: YARN-8238 [Umbrella] YARN deep learning framework examples to run on native service
• links to: 0. Submarine all document in here
• links to: Design Doc (Google Doc)