Per discussion with Chris Riccomini and Milinda Pathirage, we agreed that it seems to be a good idea to define a common representation of relational algebra on top of the operators defined in the operator layer (see
SAMZA-482), which can be the common base that we can use to generate the description/configuration of a Samza job.
This common layer can also be used by DSL-like language parser as a result of parsing a DSL program.
Some additional requirements needed in addition to pure relational algebra:
1) the common representation should include window operators and stream operators (i.e. IStream/DStream/RStream)
2) the common representation should include description on parallelism of the jobs (i.e. how many partitions the resultant Samza job will use)