Uploaded image for project: 'REEF'
  1. REEF
  2. REEF-1123

Develop a ssh-based standalone runtime

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: REEF
    • Labels:

      Description

      REEF currently has a local runtime(reef-runtime-local) for dev/test. It has served its purpose well. However, some users expressed interests in doing this in a distributed fashion.

      So, it'd be good to develop a ssh-based standalone distributed runtime. Being ssh-based means that there's less code for us to write and easy for users to deploy. Being standalone means that users don't need to set up separate resource managers(e.g., YARN, Mesos) to run REEF jobs on multiple nodes.

      Design Plan

      • scp required file to remote nodes and launch them through ssh
      • Tasks are submitted through ssh connection and run as foreground, enabling driver to receive return status
      • Evaluator logs are saved temporarily on $HOME/REEF_SSH_RUNTIME of each node, which will be sent to driver after task is finished

        Activity

        Hide
        markus.weimer Markus Weimer added a comment -

        Evaluator logs are saved temporarily on $HOME/REEF_SSH_RUNTIME of each node, which will be sent to driver after task is finished

        This should be made configurable for cases where the $HOME is size-restricted or such.

        Show
        markus.weimer Markus Weimer added a comment - Evaluator logs are saved temporarily on $HOME/REEF_SSH_RUNTIME of each node, which will be sent to driver after task is finished This should be made configurable for cases where the $HOME is size-restricted or such.
        Hide
        wsong0512 Won Wook SONG added a comment -

        Hi,
        I'm Won Wook Song, and I've left a comment also on the issue REEF-1159. I'm interested in working on the REEF project for the GSoC2016 program.
        This issue also catches my attention. This issue seems as it will help me understand REEF overall, and will help me understand and potentially solve other issues as well.
        Thanks,

        Show
        wsong0512 Won Wook SONG added a comment - Hi, I'm Won Wook Song, and I've left a comment also on the issue REEF-1159 . I'm interested in working on the REEF project for the GSoC2016 program. This issue also catches my attention. This issue seems as it will help me understand REEF overall, and will help me understand and potentially solve other issues as well. Thanks,

          People

          • Assignee:
            wonook Won Wook SONG
            Reporter:
            johnyangk John Yang
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development