[ZEPPELIN-274] Add Support for Streaming (long-running) Tasks.

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Component/s: Core, Interpreters
• Labels: streaming
• Assignee: Unassigned
• Reporter: Christian Tzolov
• Created: 01/Sep/15 22:34
• Updated: 02/Sep/15 16:29

Description

The Zeppelin's Paragraph State Machines (SM) defines 3 active states (PENDING, FINISHED and FAILED). At high level the SM looks like this:

FINISHED - (On Run) -> PENDING - (On Success) -> FINISHED /or (On Failure) -> FAILED

The Stream (long-running) processes are continuous as opposed to typical Paragraph's batch-like jobs. Currently if you start a twitter search stream task in a paragraph (by clicking the Run button) the underlying streaming platform will open a stream and will return some information for the newly created stream. The Paragraph SM will consider the job done and will transfer from PENDING into FINISHED state. In reality the actual state should be STARTED (or alike) to reflect that the stream is running and should transfer into FINISHED only after the stream is closed.

To support long-running (aka streaming) jobs Zeppelin need to extend its SM with at least one additional state: 'STARTED'.

Such improvement would benefit streaming platforms like Spark Streaming, Flink, SpringXD and alike.

Issue Links

• relates to: ZEPPELIN-275 Add SpringXD interpreter