[AIRFLOW-6963] Use low-level API for bulk insert

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.10.9
• Component/s: scheduler
• Epic Link: scheduler-performance
• Assignee: Unassigned
• Reporter: Kamil Bregula
• Created: 01/Mar/20 10:08
• Updated: 01/Mar/20 10:08

Description

Hello,
When we add a very large number of records to the database, we should use a different API.

bulk_save_objects() or bulk_insert_mappings() provide much more performance.

More information:

https://docs.sqlalchemy.org/en/13/faq/performance.html#i-m-inserting-400-000-rows-with-the-orm-and-it-s-really-slow