[SAMZA-2116] Make sendTo operators non-terminal

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2
• Assignee: Sanil Jain
• Reporter: Prateek Maheshwari
• Created: 28/Feb/19 18:09
• Updated: 20/May/19 22:20
• Resolved: 26/Apr/19 23:34

Description

sendTo(stream), sendTo(table) and sink are currently terminal operators in the High Level API. They don't return a MessageStream, and hence do not allow chaining of further operators.

Consider the following use case:

Consume a message -> send an event denoting processing start -> do some processing -> send an event denoting processing end.

Currently there's no good way to do this, since:

1. HL API does not allow chaining further operations on sendTo
2. HL API does not guarantee order of execution across multiple operators chained on the same operator.

The only way to do this today will be to use a sink function, which is not great.

One consideration when making sendTo(stream) et al. non-terminal: unlike partitionBy, which returns data in the re-keyed MessageStream partition, sendTo will return the data in the original message stream partition. This will need to be clearly called out.