Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-5623

latest_only_operator fails for schedule_interval='@once'

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.10.5
    • Fix Version/s: None
    • Component/s: operators
    • Labels:
      None

      Description

      Observation: In a dag with schedule_interval set to @once, the `latest_only_operator` fails with the following error:

       

      {{[2019-10-09 09:51:37,346] {latest_only_operator.py:48} INFO - Checking latest only with left_window: None right_window: None now: 2019-10-09 07:51:37.346697+00:00
      [2019-10-09 09:51:37,347] {models.py:1736} ERROR - '<' not supported between instances of 'NoneType' and 'datetime.datetime'
      Traceback (most recent call last):
      File "/XXXX/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/models.py", line 1633, in _run_raw_task
      result = task_copy.execute(context=context)
      File "/XXXX/anaconda/envs/airflow/lib/python3.6/site-packages/airflow/operators/latest_only_operator.py", line 51, in execute
      if not left_window < now <= right_window:
      TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
      [2019-10-09 09:51:37,363] {models.py:1756} INFO - Marking task as UP_FOR_RETRY}}

      I expected it to succeed, and allow the remainder of the dag to be ran. (if an @once dag is running, it is always the latest)

      Rootcause analysis:

      If the `schedule_interval` of a dag is `@once`, the dag's field `self._schedule_interval` is set to `None`.

      The `latest_only_operator` determines the window by passing the execution date to the dags `following_schedule()`. There the dag's `self._schedule_interval` type is compared to `six.string_types` and `timedelta`. Both type checks fail, so nothing (`None`) is returned.

      Causing the time window comparison to fail.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                goostra Gerben Oostra
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: