Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-5286

Add requeue logic to airflow scheduler and executor

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.4
    • Fix Version/s: None
    • Component/s: executors, scheduler
    • Labels:
      None

      Description

      Airflow queued tasks sometime stuck for long time without being picked up. In many cases the root cause is hard to debug. 

      The proposed solution here is to add requeue logic in airflow scheduler and executor. For tasks with a queued state in metaDB. If the task did not show in executor.queued_tasks in a certain inteval, we requeue the task so that stuck tasks can be released and picked up. The solution was used in Airbnb and proven to be helpful. 

        Attachments

          Activity

            People

            • Assignee:
              EllenWang Yingbo Wang
              Reporter:
              EllenWang Yingbo Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: