Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6769

Multiple DAGs cause infinite loop in logging

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:
      None
    • Environment:
      macOS 10.15.2
      Python 3.7.5

      Description

      After adding a second DAG to Airflow the scheduler spirals into an infinite loop and is unable to schedule any tasks. The recursion appears to have to do with logging. I've attached the output from scheduler and my requirements.txt

      If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the folder there is no issue. I don't believe it's a specific DAG as I have tried with various different test ones. I don't even have to schedule/run them, just having Airflow recognize them causes the loop. 

      I'm using MySQL and Celery as my store and worker. 

      I have also attached part of the log file for one of my DAGs outputted by the scheduler ($AIRFLOW_HOME/logs/scheduler/{dag_name}/latest/{dag_name}.log). It seems `collect_dags` is being called every second by the scheduler. 

        Attachments

        1. requirements.txt
          3 kB
          Chris Boden
        2. scheduler.log
          6 kB
          Chris Boden
        3. scheduler-dag.log
          4 kB
          Chris Boden

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cbodenaa Chris Boden
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: