Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-3607

Decreasing scheduler delay between tasks

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.10.0, 1.10.1, 1.10.2
    • Fix Version/s: 2.0.0
    • Component/s: scheduler
    • Labels:
      None
    • Environment:
      ubuntu 14.04

      Description

      I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) that says instead of checking the query for every task let the tasks report to the dagrun. I have a dag with many tasks and the delay between tasks can rise to 10 seconds or more, I already changed the configuration, added processes and memory, checked the code and did research, profiling and other experiments. I hope that this change will make a drastic change in the delay. I would be happy to discuss this solution, the research and other solutions for this issue.  

      Thanks

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                amichai07 Amichai Horvitz
                Reporter:
                amichai07 Amichai Horvitz
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 336h
                  336h
                  Remaining:
                  Remaining Estimate - 336h
                  336h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified