Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: REEF
    • Labels:
      None

      Description

      We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated in the reef-runtime-spark module, similar to the existing e.g. reef-runtime-yarn or reef-runtime-local. As a first step, we can have a Java-only solution, but later we'll need to run .NET Tasks on Executors as well.

      P.S. Here's a REEF Wiki page with more details: Spark+REEF integration

        Attachments

        1. REEF_1791_Workflow.jpg
          173 kB
          Saikat Kanjilal
        2. file-1.jpeg
          947 kB
          Saikat Kanjilal
        3. file.jpeg
          1.01 MB
          Saikat Kanjilal

          Activity

            People

            • Assignee:
              kanjilal Saikat Kanjilal
              Reporter:
              motus Sergiy Matusevych
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 1,344h
                1,344h
                Remaining:
                Remaining Estimate - 1,344h
                1,344h
                Logged:
                Time Spent - Not Specified
                Not Specified