Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-5188

Dag Bag not in-sync with DB

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.4
    • Fix Version/s: None
    • Component/s: DAG, database, scheduler, webserver
    • Labels:
      None
    • Environment:
      CentOS 7, Maria-DB, python 3.6.7, Airflow 1.10.4

      Description

      Dear all,
      we have upgraded our Airflow installation from 1.10.2 to 1.10.4 without any immidiate hick-ups. However, now, we experience problems we did not observe before the upgrade.

      When changing the START_DATE of a DAG, we also change the DAG_ID to avoid any misbehaviour of the scheduler due to changed stats of the DAG. This results in new DAGs appearing in the Dag Bag and old ones disappearing – at least it did before the upgrade.

      Now, the picture is different. I changed the START_DATE and DAG_ID of a dag, the Dag Bag lists it correctly (e.g. by manually issuing "airflow list_dags") – only the latest version is displayed, the old one is gone.
      However, the Web-UI still shows the old dag, along with the new dag. Removing it via "airflow delete_dag" does not work, prompting "Dag is still in Dag Bag, remove dag file first.". However, accessing dag details via the Web-UI fails, prompting "dag is missing", indicating that is NOT in the Dag Bag anymore (which seems to be correct).

      To complete the picture: both dags are present in the DB.
      Restarting the webserver did not help either. Restarting the scheduler has not yet been an option due to running dags. This would be my next escalation step.

      Can anyone comment on similar observations particularly with respect to having upgraded to 1.10.4?

      Cheers

      Oliver

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              oricken Oliver Ricken
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: