[AMATERASU-6] Long running actions/pipelines

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.4.0-incubating
• Fix Version/s: 0.4.0-incubating
• Flags: Important
• Assignee: Eyal Ben-Ivri
• Reporter: Yaniv Rodenski
• Created: 17/Nov/17 02:59
• Updated: 18/Feb/19 03:34

Description

As a pipeline developer, I can define pipelines and actions to be long-running, for workloads such as stream processing etc.

Example maki.yml

job-name: amaterasu-test
type: long-running
flow:

• name: action1
  type: long-running
  runner:
  group: spark
  type: scala
  file: src/file.scala
  exports:
  oddData: parquet