[AIRFLOW-6964] create_dag_run method queries individual values from the database

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.10.9
• Component/s: scheduler
• Epic Link: scheduler-performance
• Assignee: Unassigned Assign to me
• Reporter: Kamil Bregula
• Created: 01/Mar/20 10:19
• Updated: 01/Mar/20 10:19

Description

Hello,

First query:

https://github.com/apache/airflow/blob/cc562dd/airflow/jobs/scheduler_job.py#L547-L556

Second query:

https://github.com/apache/airflow/blob/cc562dd/airflow/jobs/scheduler_job.py#L596

Method A is performed for each Dag in the file. This results in the fact that we have the n + 1 problem

We should replace it with two bulk queries. This will improve performance and allow us to keep a constant number of database queries regardless of the number of DAGs in a single file.