Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2116

Make sendTo operators non-terminal

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2
    • Component/s: None
    • Labels:
      None

      Description

      sendTo(stream), sendTo(table) and sink are currently terminal operators in the High Level API. They don't return a MessageStream, and hence do not allow chaining of further operators.

      Consider the following use case:

      Consume a message -> send an event denoting processing start -> do some processing -> send an event denoting processing end.

      Currently there's no good way to do this, since:

      1. HL API does not allow chaining further operations on sendTo
      2. HL API does not guarantee order of execution across multiple operators chained on the same operator.

      The only way to do this today will be to use a sink function, which is not great.

      One consideration when making sendTo(stream) et al. non-terminal: unlike partitionBy, which returns data in the re-keyed MessageStream partition, sendTo will return the data in the original message stream partition. This will need to be clearly called out.

        Attachments

          Activity

            People

            • Assignee:
              Sanil Sanil Jain
              Reporter:
              pmaheshwari Prateek Maheshwari
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: