[KAFKA-3455] Connect custom processors with the streams DSL

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.10.0.1
• Component/s: streams
• Labels: user-experience
• Assignee: Unassigned
• Reporter: Jonathan Bender
• Created: 23/Mar/16 20:49
• Updated: 14/Jun/17 02:34
• Resolved: 05/Apr/17 19:43

Description

From the kafka users email thread, we discussed the idea of connecting custom processors with topologies defined from the Streams DSL (and being able to sink data from the processor). Possibly this could involve exposing the underlying processor's name in the streams DSL so it can be connected with the standard processor API.

Thanks for the feedback. This is definitely something we wanted to support
in the Streams DSL.

One tricky thing, though, is that some operations do not translate to a
single processor, but a sub-graph of processors (think of a stream-stream
join, which is translated to actually 5 processors for windowing / state
queries / merging, each with a different internal name). So how to define
the API to return the processor name needs some more thinking.