[SAMZA-1260] Support Control Messages Across Intermediate Streams

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.0
• Assignee: Xinyu Liu
• Reporter: Xinyu Liu
• Created: 04/May/17 16:46
• Updated: 21/Nov/17 19:34
• Resolved: 21/Nov/17 19:34

Description

In SAMZA-974, we built a mechanism to support batch job with bounded data source. The feature allows Samza jobs to shut down once all the input topic partitions reach the end of stream.

This works for applications which do not produce and consume from the same streams. With the introduction of partitionBy operators, the application can output to an intermediate stream, and then consume the same stream again for further processing. Since the end of stream tokens are not carried over from the original input streams to the intermediate streams, the job won’t be able to shut down even if all the input streams reach to the end. To address this problem, we need to extend the existing end-of-stream feature to support applications with intermediate streams.

Issue Links

• depends upon: SAMZA-1312 Add Control Messages and Intermediate Stream Serde
• depends upon: SAMZA-1321 Propagate end-of-stream messages
• depends upon: SAMZA-1322 Propagate watermark messages
• depends upon: SAMZA-1351 Separate Watermark Implementation from Propagation
• depends upon: SAMZA-1352 Detect the upstream tasks when we build the graph
• depends upon: SAMZA-1354 Pass in typed control message into EndOfStreamManager.update() and WatermarkManager.update()