Uploaded image for project: 'Apache Submarine'
  1. Apache Submarine
  2. SUBMARINE-14 [Umbrella] Support using Submarine to submit Pytorch job
  3. SUBMARINE-65

Make PyTorch support HDFS

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Do
    • None
    • None
    • TonY
    • None

    Description

      PyTorch does not support HDFS as a datasource like TensorFlow.
      Found this related issue, but nothing meaningful here: https://github.com/pytorch/pytorch/issues/5867

      I think we should make PyTorch support HDFS by contributing to PyTorch's source code.

      Here is a reference to the TF implementation of HDFS connector: https://github.com/tensorflow/tensorflow/tree/17e49b339b2b9a58ed967c69b7acb714dcd9b465/tensorflow/core/platform/hadoop

      Any other ideas to approach this problem are welcome!

      Attachments

        Activity

          People

            Unassigned Unassigned
            snemeth Szilard Nemeth
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: