Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-2729

.airflowignore is not being respected

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 1.10.0
    • Component/s: None
    • Labels:
      None

      Description

      I have a repo that in 1.10 is giving airflowignore errors that did not exist in 1.9. I have a DAG repo with the following .airflowignore:

      airflow@XXX:~$ ls -la /home/airflow/airflow/dags/airflow-tasks/
      total 172
      drwxr-xr-x 6 airflow airflow 4096 Jul 9 18:48 .
      drwxrwxr-x 3 airflow airflow 4096 Jul 9 18:48 ..
      rw-rr- 1 airflow airflow 13 Jul 9 16:20 .airflowignore

      airflow@airflow-core-i-063df3268720e58fd:~$ cat /home/airflow/airflow/dags/airflow-tasks/.airflowignore
      submodules/*

      However, the submoduled repository is being scanned for DAGs anyways, including the test suite. Note the paths in the section below:

       

      Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,814] {{models.py:351 DEBUG - Importing /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/quantflow/operators/zipline_operators.py}}
      Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,817] {{models.py:351 DEBUG - Importing /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
      Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,818] {{models.py:365 ERROR - Failed to import: /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py}}
      Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):
      Jul 09 18:52:01 airflow_web-stdout.log: File "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py", line 362, in process_file
      Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, filepath)
      Jul 09 18:52:01 airflow_web-stdout.log: File "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in load_source
      Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", line 693, in _load
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap_external>", line 665, in exec_module
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
      Jul 09 18:52:01 airflow_web-stdout.log: File "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_sqs_operators.py", line 6, in <module>
      Jul 09 18:52:01 airflow_web-stdout.log: from moto import mock_sqs
      Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 'moto'
      Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,821] {{models.py:351 DEBUG - Importing /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
      Jul 09 18:52:01 airflow_web-stdout.log: [2018-07-09 18:52:01,822] {{models.py:365 ERROR - Failed to import: /home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py}}
      Jul 09 18:52:01 airflow_web-stdout.log: Traceback (most recent call last):
      Jul 09 18:52:01 airflow_web-stdout.log: File "/home/airflow/virtualenvs/airflow/lib/python3.5/site-packages/airflow/models.py", line 362, in process_file
      Jul 09 18:52:01 airflow_web-stdout.log: m = imp.load_source(mod_name, filepath)
      Jul 09 18:52:01 airflow_web-stdout.log: File "/home/airflow/virtualenvs/airflow/lib/python3.5/imp.py", line 172, in load_source
      Jul 09 18:52:01 airflow_web-stdout.log: module = _load(spec)
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", line 693, in _load
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap_external>", line 665, in exec_module
      Jul 09 18:52:01 airflow_web-stdout.log: File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
      Jul 09 18:52:01 airflow_web-stdout.log: File "/home/airflow/airflow/dags/airflow-tasks/submodules/quantflow/tests/operators/test_zipline_operators.py", line 6, in <module>
      Jul 09 18:52:01 airflow_web-stdout.log: from freezegun import freeze_time
      Jul 09 18:52:01 airflow_web-stdout.log: ImportError: No module named 'freezegun'

       

       

        Attachments

          Activity

            People

            • Assignee:
              ash Ash Berlin-Taylor
              Reporter:
              eronarn James Meickle
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: