Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-442

Beam portability layer support for Apache Nemo

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None

      Description

      The Beam portability layer is being developed on Spark and Flink runners. Beam portability layer enables for programming in Java, Go, and Python, and enables running Beam programs programmed with the different languages on Nemo. This will be great for attracting more users to Apache Nemo. Related link: https://beam.apache.org/roadmap/portability/

      • Watch https://www.youtube.com/watch?v=I2ZqWAbbjUk for an overview and to get an idea
      • Refer to files under /runners/portability package, check requirements for the PortableRunner.java class
      • Flink is the system that provides most functionalities regarding portability, we should refer to PortableExecutionTest class, as well as FlinkPortableClientEntryPoint, FlinkPortablePipelineTranslator (for both batch and streaming), FlinkPortableRunnerResult classes, under /runners/flink, which implement the portability layer.
      • Spark runner also has points that we could refer to: PortableBatchMode,
      • We will need to make our `Nemo PortablePipelineOptions , as well as ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
      • ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
      • PipelineTranslator and PortableRunnerResult, just
      • PortableRunnerResult, just

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wonook Won Wook Song
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: