Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-274

Add Support for Streaming (long-running) Tasks.

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: Core, Interpreters
    • Labels:

      Description

      The Zeppelin's Paragraph State Machines (SM) defines 3 active states (PENDING, FINISHED and FAILED). At high level the SM looks like this:

      FINISHED - (On Run) -> PENDING - (On Success) -> FINISHED /or (On Failure) -> FAILED

      The Stream (long-running) processes are continuous as opposed to typical Paragraph's batch-like jobs. Currently if you start a twitter search stream task in a paragraph (by clicking the Run button) the underlying streaming platform will open a stream and will return some information for the newly created stream. The Paragraph SM will consider the job done and will transfer from PENDING into FINISHED state. In reality the actual state should be STARTED (or alike) to reflect that the stream is running and should transfer into FINISHED only after the stream is closed.

      To support long-running (aka streaming) jobs Zeppelin need to extend its SM with at least one additional state: 'STARTED'.

      Such improvement would benefit streaming platforms like Spark Streaming, Flink, SpringXD and alike.

        Attachments

        1. ZeppelinSMD.png
          20 kB
          Christian Tzolov

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tzolov Christian Tzolov
              • Votes:
                5 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated: