Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
On liminal start,
When a wrong property is defined, it fails to load the entire yml.
One of two options:
- Start should not fail to load yml, just write a warning.
- A validity check should be way sooner (liminal build)
liminal-webserver | Registering DAG from path: /opt/airflow/dags/pipelines liminal-webserver | /opt/airflow/dags/pipelines liminal-webserver | /opt/airflow/dags/pipelines/liminal.yml liminal-webserver | found 1 in path: /opt/airflow/dags/pipelines liminal-webserver | Registering DAG for file: /opt/airflow/dags/pipelines/liminal.yml liminal-webserver | [2020-11-17 19:40:37,089] {dagbag.py:259} ERROR - Failed to import: /opt/airflow/dags/liminal_dags.py liminal-webserver | Traceback (most recent call last): liminal-webserver | File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 256, in process_file liminal-webserver | m = imp.load_source(mod_name, filepath) liminal-webserver | File "/usr/local/lib/python3.6/imp.py", line 172, in load_source liminal-webserver | module = _load(spec) liminal-webserver | File "<frozen importlib._bootstrap>", line 684, in _load liminal-webserver | File "<frozen importlib._bootstrap>", line 665, in _load_unlocked liminal-webserver | File "<frozen importlib._bootstrap_external>", line 678, in exec_module liminal-webserver | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed liminal-webserver | File "/opt/airflow/dags/liminal_dags.py", line 119, in <module> liminal-webserver | register_dags(environment.get_dags_dir()) liminal-webserver | File "/opt/airflow/dags/liminal_dags.py", line 80, in register_dags liminal-webserver | dag, pipeline['pipeline'], parent if parent else None, task, trigger_rule liminal-webserver | File "/home/airflow/.local/lib/python3.6/site-packages/liminal/runners/airflow/tasks/python.py", line 39, in __init__ liminal-webserver | self.input_path = self.config['input_path'] liminal-webserver | KeyError: 'input_path'