Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-7090

With depends_on_past=True, second instance of task not scheduled even when first instance ran successfully

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersLinkUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:
      None

      Description

      The first task of my DAG has `depends_on_past=True` and `wait_for_downstream=True`. The DAG ran automatically when I turned it `On` and it (DAG and all tasks) completed successfully. Now, I manually triggered the DAG again (after the first run completed successfully), but this time, my first task did not start running. `Task Instance Details` for this task shows `depends_on_past is true for this task's DAG, but the previous task instance has not run yet.`

      According to docs about `depends_on_past (boolean)`:

      > when set to True, keeps a task from getting triggered if the previous
      > schedule for the task hasn’t succeeded.

      The first DAG run was successful and the first instance of the first task was (obviously) successful as well. Yet, why is the second instance of the first task complaining that the `previous task instance has not run yet`?

      Relevant parts of my code:

       

      ...
       args = { 'owner': 'USC Graduate School', 'start_date': days_ago(1), }
      dag = DAG(
       dag_id='enrollment_import_poc',
       default_args=args,
       schedule_interval='0 0 * * *',
       dagrun_timeout=timedelta(minutes=60),
       max_active_runs=1,
       template_searchpath = os.environ.get('AIRFLOW_HOME'),
       tags=['uscgradschool']
       )
      schools = MsSqlOperator(
       task_id='schools',
       depends_on_past=True,
       wait_for_downstream=True,
       sql=os.path.join("queries", "01_schools.sql"),
       mssql_conn_id="mssql_local",
       autocommit=True,
       dag=dag
       )
       ...
      

        Attachments

        1. 1.png
          96 kB
          Abhilash Kishore
        2. 2.png
          70 kB
          Abhilash Kishore
        3. 3.png
          110 kB
          Abhilash Kishore
        4. 4.png
          97 kB
          Abhilash Kishore

          Activity

            People

            • Assignee:
              Unassigned Assign to me
              Reporter:
              abhilash1in Abhilash Kishore

              Dates

              • Created:
                Updated:

                Issue deployment