Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8135

Hadoop {Submarine} Project: Simple and scalable deployment of deep learning training / serving jobs on Hadoop

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Description:

      Goals:

      • Allow infra engineer / data scientist to run unmodified Tensorflow jobs on YARN.
      • Allow jobs easy access data/models in HDFS and other storages.
      • Can launch services to serve Tensorflow/MXNet models.
      • Support run distributed Tensorflow jobs with simple configs.
      • Support run user-specified Docker images.
      • Support specify GPU and other resources.
      • Support launch tensorboard if user specified.
      • Support customized DNS name for roles (like tensorboard.$user.$domain:6006)

      Why this name?

      • Because Submarine is the only vehicle can let human to explore deep places. B-)

      Please refer to on-going design doc, and add your thoughts: https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                leftnoteasy Wangda Tan
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                0 Vote for this issue
                Watchers:
                25 Start watching this issue

                Dates

                • Created:
                  Updated: