[GSOC-240] [GSoC][Beam] Advancing the Beam-on-Ray runner

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: Beam full-time gsoc gsoc2023 mentor
• Assignee: Unassigned
• Reporter: Pablo Estrada
• Created: 21/Mar/23 15:25
• Updated: 22/Mar/23 03:41

Description

There is a community effort to build a Beam runner to run Beam pipelines on top of Ray: https://github.com/ray-project/ray_beam_runner/

 

This involves pushing that project forward. It will require writing lots of Python code, and specifically going through the list of issues (https://github.com/ray-project/ray_beam_runner/issues) and solving as many of them as possible to make sure the runner is compliant.

Good resource docs:

• https://docs.google.com/document/d/1vt78s48Q0aBhaUCHrVrTUsProJSP8-EBqDDRGTPEr0Y/edit#heading=h.bhgd35otqjly
• https://docs.google.com/document/d/1k5JpMzIP-l7BssLpp6-gSNR-796ttZcACaY1p2CEFho/edit#heading=h.f5fgmvcor6lo 

This project is large.

Issue Links

• is a clone of: GSOC-111 [GSoC][Beam] An IntelliJ plugin to develop Apache Beam pipelines and the Apache Beam SDKs
• is cloned by: GSOC-241 [GSoC][Beam] Advancing the Rust SDK on Beam