[GEARPUMP-66] Investigate on Akka-persistence as a general storage implementation

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Manu Zhang
• Created: 21/Apr/16 07:46
• Updated: 21/Apr/16 07:48

Description

imported from https://github.com/gearpump/gearpump/issues/1822 on behalf of clockfly,

Motive:
Currently we use replicated memory to store some data, it has risks when the whole cluster is brought down due to power failure.

We need to investigate wether Akka-persistence can be used as a distributed, pluggable, general storage to persist data like:

1. application timestamp
2. application submission info (like jar, and confgs...)
3. application runtime storage for KV.

What is the expected output for this investigation:
We expect to see a test project in scala which use "akka persistence", it should:

1. provide a API to read/write large chunk of data in a streaming way, suggest to test it on Tested on HDFS, and Redis.
2. connect with akka-data-replication to persist the CRDT data, and why this can still be consistent?
3. Demo how this can be plugged, how to switch between HDFS and Redis. What is the new classpath introduced for each storage(HDFS or Redis)?

Issue Links

• is related to: GEARPUMP-63 Gearpump Storage framework