Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-3516

Improve K8 Executors Pod creation Rate

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Resolved
    • Affects Version/s: 1.10.1
    • Fix Version/s: 1.10.3
    • Component/s: operators
    • Labels:

      Description

      It seems that Airflow scheduler is creating worker pod sequentially(one pod per scheduler loop) in Kubernetes executors sync function which is called from base executors.heartbeat() function.

      One enhancement could be to create the k8 pod in batches where batch size can be configured through airflow.cfg file

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ramandumcs raman
                Reporter:
                ramandumcs raman
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: