[OOZIE-2988] Investigate Apache Airflow

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: client
• Assignee: Daniel Becker
• Reporter: Andras Piros
• Created: 14/Jul/17 08:37
• Updated: 31/Jul/17 09:07
• Resolved: 17/Jul/17 12:18

Description

Investigate current version of Apache Airflow regarding following:

• how to define a DAG
  • nodes
  • edges
  • transitions
  • decision and fork / join nodes
  • checks on DAGness, and on completeness
• how are connections between DAGs handled
  • state of one running / already finished DAG is respected by another DAG
• creation / running process
  • what to define w/ Python code
  • and what using templates
  • inheritance in Python
  • third party libs in Python code while creating a DAG
  • how are DAGs stored and submitted / run
  • what patterns are used under the hood (builder, delegate, ...)
  • how is logic considered (Oozie EL functions)
• how to schedule a DAG (Oozie Coordinators)
• how to bundle these together (Oozie Bundles)