Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-1488

Add a sensor operator to wait on DagRuns

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: contrib, operators
    • Labels:
      None

      Description

      The ExternalTaskSensor operator already allows for encoding dependencies on tasks in external DAGs. However, when you have teams, each owning multiple small-to-medium sized DAGs, it is desirable to be able to wait on an external DagRun as a whole. This allows the owners of an upstream DAG to refactor their code freely by splitting/squashing task responsibilities, without worrying about dependent DAGs breaking.

      I'll now enumerate the easiest ways of achieving this that come to mind:

      • Make all DAGs always have a join DummyOperator in the end, with a task id that follows some convention, e.g., "{{ dag_id }}._end_".
      • Make ExternalTaskSensor poke for a DagRun instead of TaskInstances when the external_task_id argument is None.
      • Implement a separate DagRunSensor operator.

      After considerations, we decided to implement a separate operator, which we've been using in the team for our workflows, and I think it would make a good addition to contrib.

        Attachments

          Activity

            People

            • Assignee:
              ysagade Yati
              Reporter:
              ysagade Yati
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: