[IGNITE-12228] Implement an Apache Beam runner on top of Ignite compute grid?

• Type: Wish
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Saikat Maitra
• Reporter: Romain Manni-Bucau
• Created: 24/Sep/19 16:59
• Updated: 15/Aug/20 19:35

Description

Apache Ignite provides a compute grid.

It is therefore feasible to use that to execute a DAG pipeline.

Therefore implementing a Apache Beam runner executing a DAG on an ignite cluster would be very valuable for users and would provide a very interesting alternative to Spark, Dataflow etc...which would be very valuable when using data from ignite itself. 

Hoping it can help, here is Hazelcast Jet implementation which is not that far even if Jet is a bit more advanced in term of DAG support: https://github.com/apache/beam/tree/master/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet

Issue Links

• duplicates: IGNITE-7198 Integrate with Apache Beam