Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6853

ExternalTaskSensor check_existence does not work with packaged DAGs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: operators
    • Labels:
      None

      Description

      The current implementation of ExternalTaskSensor uses the file location of a DAG to determine if it still exists. This is realized by the following code fragment:

                      if not os.path.exists(dag_to_wait.fileloc):
                          raise AirflowException('The external DAG '
                                                 '{} was deleted.'.format(self.external_dag_id)
      

      In case the DAG to check is part of a Zip archive, fileloc looks something like: /usr/local/airflow/dags/analytics.zip/create_keyframes.py. This can obviously not be checked with os.path.exists and therefore the sensor is currently unable to handle packaged DAGs at all.

      This also contradicts the explicit note in the DAG model:

          # The location of the file containing the DAG object
          # Note: Do not depend on fileloc pointing to a file; in the case of a
          # packaged DAG, it will point to the subpath of the DAG within the
          # associated zip.
      

      I suspect, the sensor should instead use is_active.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              languitar Johannes Wienke
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: