Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-5554

Airflow 1.10.4+ needs statsd>=3.3.0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.10.4, 1.10.5
    • Fix Version/s: 1.10.6
    • Component/s: DagRun, dependencies
    • Labels:
      None

      Description

      Our environment configuration uses statsd 3.2.2, which is a valid dependency version choice in airflow's requirements. However if statsd is enabled in airflow config, there is a critical bug preventing dag runs from completing properly when a timedelta is passed to the statsd timing to be formatted into a float:

      https://github.com/apache/airflow/blob/66a139d734bd434caa792007c3b980ca4cf8f931/airflow/models/dagrun.py#L354

      Causing an exception to be raised due to the unexpected delta data type:

      File "/opt/evidation/data-processing-pipeline/lib/python3.5/site-packages/airflow/jobs/scheduler_job.py", line 1547, in process_file
      self._process_dags(dagbag, dags, ti_keys_to_schedule)
      File "/opt/evidation/data-processing-pipeline/lib/python3.5/site-packages/airflow/jobs/scheduler_job.py", line 1242, in _process_dags
      schedule_delay)
      File "/opt/evidation/data-processing-pipeline/lib/python3.5/site-packages/statsd/client.py", line 93, in timing
      self._send_stat(stat, '%0.6f|ms' % delta, rate)
      TypeError: a float is required

      The code to detect the timedelta type and convert to millis was introduced in 3.3.0:

      https://github.com/jsocol/pystatsd/blob/1c90b9fdf322680e2625da659abc2aa5d79b5bff/statsd/client/base.py#L28

      So the setup.py in airflow should be updated to make this the minimum dependency:

      https://github.com/apache/airflow/blob/66a139d734bd434caa792007c3b980ca4cf8f931/setup.py#L262

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                suburbanmtman David Smith
                Reporter:
                suburbanmtman David Smith
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: