[BAHIR-135] Add Spark Streaming Hazelcast Extension

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark Streaming Connectors
• Assignee: Unassigned
• Reporter: Eren Avsarogullari
• Created: 09/Sep/17 20:01
• Updated: 23/Sep/17 11:56

Description

I would like to propose Spark Streaming Hazelcast extension.

Hazelcast is an in-memory data grid(IMDG) solution under Apache 2 License and provides distributed data structures such as distributed map, list, set, queue (etc). When a new entry is added, updated, removed or evicted, a new event is fired by Hazelcast. This flow is almost same for above all distributed data structures. This extension aims to subscribe these distributed events via Hazelcast Event Listeners and create a DStream in the light of distributed data structure changes. This extension supports Distributed Map, List, Set, Queue, Topic, MultiMap and Replicated Map.

Please find the following documentation for further details.
Proposal: https://docs.google.com/document/d/1YN_9u72Wv699g8ivM3c8K_zZUbUl73JtquWy-g71Tm4/edit?usp=sharing

Also repo is ready for review. It covers implementation, full unit test coverage and examples as well.
Repo: https://github.com/erenavsarogullari/bahir/tree/Hazelcast_Streaming

This extension can be useful for both Spark and Hazelcast communities to listen these Hazelcast events & analyze them and transform the events payloads via Spark.

Please let me know if you need further details and all feedbacks are welcome in advance.