Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-2895

Prevent scheduler from spamming heartbeats/logs

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.10.1
    • Component/s: scheduler
    • Labels:
      None

      Description

      There seems to be a couple of problems with https://github.com/apache/incubator-airflow/pull/2986 that cause the sleep to not trigger and Scheduler heartbeating/logs to be spammed:

      1. If all of the files are being processed in the queue, there is no sleep (can be fixed by sleeping for min_sleep even if there are no files)
      2. I have heard reports that some files can return a parsing time that is monotonically increasing for some reason (e.g. file actually parses in 1s each loop, but the reported duration seems to use the very time the file was parsed as the start time instead of the last time), I haven't confirmed this but it sounds problematic.

      To unblock the release I'm reverting this PR for now. It should be re-added with tests/mocking.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                aoen Dan Davydov
                Reporter:
                aoen Dan Davydov
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: