Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-442

Beam portability layer support for Apache Nemo

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None

    Description

      The Beam portability layer is being developed on Spark and Flink runners. Beam portability layer enables for programming in Java, Go, and Python, and enables running Beam programs programmed with the different languages on Nemo. This will be great for attracting more users to Apache Nemo. Related link: https://beam.apache.org/roadmap/portability/

      • Watch https://www.youtube.com/watch?v=I2ZqWAbbjUk for an overview and to get an idea
      • Refer to files under /runners/portability package, check requirements for the PortableRunner.java class
      • Flink is the system that provides most functionalities regarding portability, we should refer to PortableExecutionTest class, as well as FlinkPortableClientEntryPoint, FlinkPortablePipelineTranslator (for both batch and streaming), FlinkPortableRunnerResult classes, under /runners/flink, which implement the portability layer.
      • Spark runner also has points that we could refer to: PortableBatchMode,
      • We will need to make our `Nemo PortablePipelineOptions , as well as ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
      • ClientEntryPoint, PipelineTranslator and PortableRunnerResult, just
      • PipelineTranslator and PortableRunnerResult, just
      • PortableRunnerResult, just

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            wonook Wonook

            Dates

              Created:
              Updated:

              Slack

                Issue deployment