Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6964

create_dag_run method queries individual values from the database

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:
      None

      Description

      Hello,

      First query:

      https://github.com/apache/airflow/blob/cc562dd/airflow/jobs/scheduler_job.py#L547-L556

      Second query:

      https://github.com/apache/airflow/blob/cc562dd/airflow/jobs/scheduler_job.py#L596

      Method A is performed for each Dag in the file. This results in the fact that we have the n + 1 problem

      We should replace it with two bulk queries. This will improve performance and allow us to keep a constant number of database queries regardless of the number of DAGs in a single file.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kamil.bregula Kamil Bregula
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: