Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6859

Airflow scheduler looking for logs of example DAGs in the wrong folder

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Invalid
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:
      None

      Description

      My AIRFLOW_HOME is set to /home/airflow

      I am running Airflow as a service

      `cat /etc/systemd/system/airflow-scheduler.service` :

       

      [Unit]
      Description=Airflow scheduler daemon
      After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service mssql-server.service
      Wants=postgresql.service mysql.service redis.service rabbitmq-server.service mssql-server.service
      [Service]
      EnvironmentFile=/etc/sysconfig/airflow
      User=airflow
      Group=airflow
      Type=simple
      ExecStart=/usr/local/bin/airflow scheduler
      Restart=always
      RestartSec=5s
      [Install]
      WantedBy=multi-user.target
      

       

       

      BUG:

      Airflow scheduler looking for logs of example DAGs in the wrong folder. Example:

       

      Feb 21 01:57:24 airflow-poc airflow[25847]: Process DagFileProcessor281-Process:
      Feb 21 01:57:24 airflow-poc airflow[25847]: Traceback (most recent call last):
      Feb 21 01:57:24 airflow-poc airflow[25847]:   File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
      Feb 21 01:57:24 airflow-poc airflow[25847]:     self.run()
      Feb 21 01:57:24 airflow-poc airflow[25847]:   File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
      Feb 21 01:57:24 airflow-poc airflow[25847]:     self._target(*self._args, **self._kwargs)
      Feb 21 01:57:24 airflow-poc airflow[25847]:   File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 135, in _run_file_processor
      Feb 21 01:57:24 airflow-poc airflow[25847]:     set_context(log, file_path)
      Feb 21 01:57:24 airflow-poc airflow[25847]:   File "/usr/local/lib/python3.6/dist-packages/airflow/utils/log/logging_mixin.py", line 198, in set_context
      Feb 21 01:57:24 airflow-poc airflow[25847]:     handler.set_context(value)
      Feb 21 01:57:24 airflow-poc airflow[25847]:   File "/usr/local/lib/python3.6/dist-packages/airflow/utils/log/file_processor_handler.py", line 66, in set_context
      Feb 21 01:57:24 airflow-poc airflow[25847]:     self.handler = logging.FileHandler(local_loc)
      Feb 21 01:57:24 airflow-poc airflow[25847]:   File "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__
      Feb 21 01:57:24 airflow-poc airflow[25847]:     StreamHandler.__init__(self, self._open())
      Feb 21 01:57:24 airflow-poc airflow[25847]:   File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open
      Feb 21 01:57:24 airflow-poc airflow[25847]:     return open(self.baseFilename, self.mode, encoding=self.encoding)
      Feb 21 01:57:24 airflow-poc airflow[25847]: PermissionError: [Errno 13] Permission denied: '/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log'

       

       

      The same is happening for all example DAGs (haven't loaded any other DAGs yet).

       

      Airflow scheduler is looking for the logs of the example DAG in `/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`. It should instead look in `/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`  instead (without the '/home/airflow' prefix).

       

      In the `dag` table in the database, the DAG with dag_id `example_skip_dag.` has fileloc `/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py`.

       

      Not sure why Airflow scheduler is appending `/home/airflow' to this path

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              abhilash1in Abhilash Kishore
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: