-
Type:
Bug
-
Status: Resolved
-
Priority:
Blocker
-
Resolution: Invalid
-
Affects Version/s: 1.10.9
-
Fix Version/s: None
-
Component/s: database
-
Labels:None
-
Environment:Centos 6
-
Flags:Important
I am trying to execute airflow with MS SQL Server as the backend.
Eventhough airflow initdb works fine, airflow resetdb throws an error for which I have already created an issue.
After airflow initdb, with some dags in the dags folder, if I run the airflow scheduler, then I am getting the following error and it fails for every dag and all the time. I have tried with different (including latest) pyodbc versions
Please find below the error.
ERROR - Process timed out, PID: 94590
[2020-02-15 02:30:52,094] {celery_executor.py:266} ERROR - Error fetching Celery task state, ignoring it:StatementError("(builtins.SystemError) <class 'pyodbc.Error'> returned a result with an error set",)
Celery Task ID: ('simple_dag', 'print_date0', datetime.datetime(2020, 2, 14, 23, 0, tzinfo=<Timezone [UTC]>), 1)
Traceback (most recent call last):
File "/home/ops/.pyenv/versions/3.6.10/lib/python3.6/encodings/utf_16_le.py", line 15, in decode
def decode(input, errors='strict'):
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/airflow/utils/timeout.py", line 43, in handle_timeout
raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 94590
The above exception was the direct cause of the following exception:
airflow.exceptions.AirflowTaskTimeout: decoding with 'utf-16le' codec failed (AirflowTaskTimeout: Timeout, PID: 94590)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1318, in _cursor_execute
self.dialect.do_execute(cursor, statement, parameters, context)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
SystemError: <class 'pyodbc.Error'> returned a result with an error set
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 108, in fetch_celery_task_state
res = (celery_task[0], celery_task[1].state)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/result.py", line 475, in state
return self._get_task_meta()['status']
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/result.py", line 414, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/base.py", line 451, in get_task_meta
meta = self._get_task_meta_for(task_id)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/_init_.py", line 51, in _inner
return fun(*args, **kwargs)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/_init_.py", line 151, in _get_task_meta_for
session = self.ResultSession()
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/_init_.py", line 110, in ResultSession
**self.engine_options)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
self.prepare_models(engine)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/session.py", line 54, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4316, in create_all
ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2048, in _run_visitor
with self._optional_conn_ctx_manager(connection) as conn:
File "/home/ops/.pyenv/versions/3.6.10/lib/python3.6/contextlib.py", line 81, in _enter_
return next(self.gen)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2040, in _optional_conn_ctx_manager
with self._contextual_connect() as conn:
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2242, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
return fn()
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
return _ConnectionFairy._checkout(self)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 774, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
self._dec_overflow()
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in _exit_
compat.reraise(exc_type, exc_value, exc_tb)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
raise value
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
return self._create_connection()
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in _init_
self.__connect(first_connect_check=True)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 662, in __connect
).exec_once_unless_exception(self.connection, self)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 314, in exec_once_unless_exception
self._exec_once_impl(True, *args, **kw)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 285, in _exec_once_impl
self(*args, **kw)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 322, in _call_
fn(*args, **kw)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 1485, in go
return once_fn(*arg, **kw)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 199, in first_connect
dialect.initialize(c)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/dialects/mssql/base.py", line 2378, in initialize
super(MSDialect, self).initialize(connection)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 318, in initialize
self.returns_unicode_strings = self._check_unicode_returns(connection)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 415, in _check_unicode_returns
results = {check_unicode(test) for test in tests}
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 415, in <setcomp>
results = {check_unicode(test) for test in tests}
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 384, in check_unicode
connection._cursor_execute(cursor, statement, parameters)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1321, in _cursor_execute
e, statement, parameters, cursor, context
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1318, in _cursor_execute
self.dialect.do_execute(cursor, statement, parameters, context)
File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.StatementError: (builtins.SystemError) <class 'pyodbc.Error'> returned a result with an error set
[SQL: SELECT CAST('test unicode returns' AS NVARCHAR(60)) AS anon_1]