Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6748

Tasks fail to run when alternate AIRFLOW_HOME is defined.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.7
    • Fix Version/s: None
    • Component/s: configuration, DagRun
    • Labels:
      None
    • Environment:
      ubuntu 16.04

      Description

      When an alternate AIRFLOW_HOME is defined via env var and a DAG is triggered either by a schedule or manually triggered the tasks fail to execute. This occurs because the alternate AIRFLOW_HOME is ignored and a default airflow.cfg is generated at execution time in ~/airflow. Which contains all the incorrect settings, causing the DAG to fail.

       

      My setup:

      DaskExecutor

      Postgres Metadata DB

      S3 remote logging

       

      What I see in my dask worker log:

      SequentialExecutor being used

      Failed to write logs to SQLITE

       

      Work around:

      Copying alternate $AIRFLOW_HOME/airflow.cfg to ~/airflow/airflow.cfg

      This insures a default airflow.cfg is not generated in ~/airflow with incorrect settings...

      My DAG then runs successfully

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ehlavaty Evan Hlavaty
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: