Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-274

Add Support for Streaming (long-running) Tasks.


    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: Core, Interpreters
    • Labels:


      The Zeppelin's Paragraph State Machines (SM) defines 3 active states (PENDING, FINISHED and FAILED). At high level the SM looks like this:

      FINISHED - (On Run) -> PENDING - (On Success) -> FINISHED /or (On Failure) -> FAILED

      The Stream (long-running) processes are continuous as opposed to typical Paragraph's batch-like jobs. Currently if you start a twitter search stream task in a paragraph (by clicking the Run button) the underlying streaming platform will open a stream and will return some information for the newly created stream. The Paragraph SM will consider the job done and will transfer from PENDING into FINISHED state. In reality the actual state should be STARTED (or alike) to reflect that the stream is running and should transfer into FINISHED only after the stream is closed.

      To support long-running (aka streaming) jobs Zeppelin need to extend its SM with at least one additional state: 'STARTED'.

      Such improvement would benefit streaming platforms like Spark Streaming, Flink, SpringXD and alike.


        1. ZeppelinSMD.png
          20 kB
          Christian Tzolov

          Issue Links



              • Assignee:
                tzolov Christian Tzolov
              • Votes:
                5 Vote for this issue
                11 Start watching this issue


                • Created: