Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.12.0
-
None
-
None
Description
Samza today supports RocksDB and MemDB as local data stores, which enables users to cache data for later usage during stream processing. However, the population of a data store is end user’s responsibility. This introduced additional effort from end user to develop for and maintain data stores, and also deal with corner cases such as reload after consumers falling off.
We want to have an adjunct data (AD) store that is a read-only cache. It automatically stores streaming data for later usage. Adjunct data can be accessed the same way as accessing a key-value store in Samza. Data can be either partitioned or unpartitioned. If the dataset is small enough to fit in a RocksDB instance, the same copy would be populated in every container via a broadcast stream; if it is large enough fit in one database instance it would be partitioned across containers of a Samza job.
A dataset delivered in a stream can be either bounded or unbounded, an example of an unbounded dataset could be a database change stream, and an example of a bounded dataset could be the content of a file. When Samza is running in 24x7 mode, the stream for a bounded dataset may deliver multiple versions.
This proposal focuses on unbounded datasets.
The SEP can be found here
https://cwiki.apache.org/confluence/display/SAMZA/Adjunct+Data+Store+for+Unbounded+DataSets