Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-4018

[Umbrella] Workflow and orchestration

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      Zeppelin is now only able to support corntab. A note is executed periodically at a specified time.

      In the actual operating environment, The way through corntab is too simple, Workflow orchestration for paragraphs of different interpreters in multiple notes (or a note) in a specific execution order cannot be supported.

      We created a lot of notes in our zeppelin, We urgently need zeppelin to support the layout of the workflow. This can form a closed loop of data processing. Not just an interactive development tool.

      Especially in machine learning, Because machine learning generally has a long task execution.
      A typical example is as follows:
      1) First, obtain data from HDFS through spark;
      2) Clean and convert the data through sparksql;
      3) Feature extraction of data through spark;
      4) Tensorflow writing algorithm through hadoop submarine;
      5) Distribute the tensorflow algorithm as a job to YARN or k8s for batch processing;
      6) Publish the training acquisition model and provide online prediction services;
      7) Model prediction by flink;
      8) Receive incremental data through flink for incremental update of the model;

      Therefore, zeppelin is especially required to have the ability to arrange workflows.

      Please refer to on-going design doc, and add your thoughts:

      https://docs.google.com/document/d/1pQjVifOC1knPBuw3LVvby7GyNDXaeBq1ltRg6x4vDxM/edit?usp=sharing 

      Attachments

        1. workflow-graph.png
          193 kB
          Xun Liu
        2. workflow-executions.png
          51 kB
          Xun Liu
        3. workflow-configure.png
          224 kB
          Xun Liu

        Issue Links

          Activity

            liuxun323 Xun Liu added a comment -

            Hello, everyone, I have completed the workflow system design, please review, you can directly modify the document or fill in the comments.

            liuxun323 Xun Liu added a comment - Hello, everyone, I have completed the workflow system design, please review, you can directly modify the document or fill in the comments.

            Hi liuxun323  . I am a student at RV College of Engineering and interested to take this feature as my project for GSOC 2019.

            Can you guide me with the proceedings.

            khamar KHAMAR ALI SHAIKH added a comment - Hi  liuxun323   . I am a student at RV College of Engineering and interested to take this feature as my project for GSOC 2019. Can you guide me with the proceedings.
            liuxun323 Xun Liu added a comment -

            khamar, Please assign the https://issues.apache.org/jira/browse/ZEPPELIN-4076  to yourself. 

            liuxun323 Xun Liu added a comment - khamar , Please assign the https://issues.apache.org/jira/browse/ZEPPELIN-4076   to yourself. 

            Hi Xun Liu , i want to take this project for GSOC 2019 , can you please
            help with this?

            –
            Thank you and Regards,

            Khamar Ali Shaikh

            khamar KHAMAR ALI SHAIKH added a comment - Hi Xun Liu , i want to take this project for GSOC 2019 , can you please help with this? – Thank you and Regards, Khamar Ali Shaikh
            liuxun323 Xun Liu added a comment -

            khamar, Now you have permission assign https://issues.apache.org/jira/browse/ZEPPELIN-4076 ticket to yourself.

            liuxun323 Xun Liu added a comment - khamar , Now you have permission assign https://issues.apache.org/jira/browse/ZEPPELIN-4076  ticket to yourself.
            hghosh777 HOMAGNI GHOSH added a comment -

            I am HOMAGNI GHOSH currently studying computer science and engineering.I want to contribute in this GSOC project. Please help with further proceedings.

            hghosh777 HOMAGNI GHOSH added a comment - I am HOMAGNI GHOSH currently studying computer science and engineering.I want to contribute in this GSOC project. Please help with further proceedings.
            liuxun323 Xun Liu added a comment -

            Hi, hghosh777, You can look at the workflow design documentation first. Accurate understanding of each module, It is the first step in our work.

            Take a look at which modules in the design document are of interest to you and can be done, You can do this by commenting directly in the document.

            Please send me an email: liuxun323@gmail.com. This facilitates the connection between us at all times.

            liuxun323 Xun Liu added a comment - Hi, hghosh777 , You can look at the workflow design documentation first. Accurate understanding of each module, It is the first step in our work. Take a look at which modules in the design document are of interest to you and can be done, You can do this by commenting directly in the document. Please send me an email: liuxun323@gmail.com. This facilitates the connection between us at all times.
            liuxun323 Xun Liu added a comment -

            zjffdu, Because the workflow is multiple people, Develop by dividing modules, How do we submit the code?
            Is it a development branch of workflow created in zeppelin?
            For our code submission merge, I will periodically rebase master.
            After the workflow is developed,Rearrange the code and merge it into the master branch?

            liuxun323 Xun Liu added a comment - zjffdu , Because the workflow is multiple people, Develop by dividing modules, How do we submit the code? Is it a development branch of workflow created in zeppelin? For our code submission merge, I will periodically rebase master. After the workflow is developed,Rearrange the code and merge it into the master branch?

            liuxun323
            Myself Mukul pursuing master's in computer science to develop skillset on machine learning and related toolkit would love to contribute to Zeppelin project through GSOC 2019 umbrella. I have had couple of years of experience in developing java APIs using streams

            Please let me know of any details I should consider before submitting a formal proposal

            vyomkesh Mukul chand yadav added a comment - liuxun323 Myself Mukul pursuing master's in computer science to develop skillset on machine learning and related toolkit would love to contribute to Zeppelin project through GSOC 2019 umbrella. I have had couple of years of experience in developing java APIs using streams Please let me know of any details I should consider before submitting a formal proposal
            daizhixia zhixia dai added a comment -

            hello, liuxun323 is the job started?
            Thanks

            daizhixia zhixia dai added a comment - hello, liuxun323 is the job started? Thanks

            People

              liuxun323 Xun Liu
              liuxun323 Xun Liu
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: