Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6814

Airflow Scheduler fails for any dag with MS SQL server as the backend

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Invalid
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: database
    • Labels:
      None
    • Environment:
      Centos 6
    • Flags:
      Important

      Description

      I am trying to execute airflow with MS SQL Server as the backend. 
      Eventhough airflow initdb works fine, airflow resetdb throws an error for which I have already created an issue. 

      After airflow initdb, with some dags in the dags folder, if I run the airflow scheduler, then I am getting the following error and it fails for every dag and all the time. I have tried with different (including latest) pyodbc versions

      Please find below the error. 

      ERROR - Process timed out, PID: 94590

      [2020-02-15 02:30:52,094] {celery_executor.py:266} ERROR - Error fetching Celery task state, ignoring it:StatementError("(builtins.SystemError) <class 'pyodbc.Error'> returned a result with an error set",)
      Celery Task ID: ('simple_dag', 'print_date0', datetime.datetime(2020, 2, 14, 23, 0, tzinfo=<Timezone [UTC]>), 1)

      Traceback (most recent call last):

      File "/home/ops/.pyenv/versions/3.6.10/lib/python3.6/encodings/utf_16_le.py", line 15, in decode
      def decode(input, errors='strict'):
      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/airflow/utils/timeout.py", line 43, in handle_timeout
      raise AirflowTaskTimeout(self.error_message)

      airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 94590

      The above exception was the direct cause of the following exception:

      airflow.exceptions.AirflowTaskTimeout: decoding with 'utf-16le' codec failed (AirflowTaskTimeout: Timeout, PID: 94590)

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1318, in _cursor_execute
      self.dialect.do_execute(cursor, statement, parameters, context)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
      cursor.execute(statement, parameters)

      SystemError: <class 'pyodbc.Error'> returned a result with an error set

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 108, in fetch_celery_task_state
      res = (celery_task[0], celery_task[1].state)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/result.py", line 475, in state
      return self._get_task_meta()['status']

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/result.py", line 414, in _get_task_meta
      return self._maybe_set_cache(self.backend.get_task_meta(self.id))

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/base.py", line 451, in get_task_meta
      meta = self._get_task_meta_for(task_id)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/_init_.py", line 51, in _inner
      return fun(*args, **kwargs)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/_init_.py", line 151, in _get_task_meta_for
      session = self.ResultSession()

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/_init_.py", line 110, in ResultSession
      **self.engine_options)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
      self.prepare_models(engine)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/celery/backends/database/session.py", line 54, in prepare_models
      ResultModelBase.metadata.create_all(engine)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4316, in create_all
      ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2048, in _run_visitor
      with self._optional_conn_ctx_manager(connection) as conn:

      File "/home/ops/.pyenv/versions/3.6.10/lib/python3.6/contextlib.py", line 81, in _enter_
      return next(self.gen)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2040, in _optional_conn_ctx_manager
      with self._contextual_connect() as conn:

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2242, in _contextual_connect
      self._wrap_pool_connect(self.pool.connect, None),

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
      return fn()

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
      return _ConnectionFairy._checkout(self)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 774, in _checkout
      fairy = _ConnectionRecord.checkout(pool)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
      rec = pool._do_get()

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
      self._dec_overflow()

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in _exit_
      compat.reraise(exc_type, exc_value, exc_tb)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
      raise value

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
      return self._create_connection()

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
      return _ConnectionRecord(self)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in _init_
      self.__connect(first_connect_check=True)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 662, in __connect
      ).exec_once_unless_exception(self.connection, self)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 314, in exec_once_unless_exception
      self._exec_once_impl(True, *args, **kw)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 285, in _exec_once_impl
      self(*args, **kw)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 322, in _call_
      fn(*args, **kw)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 1485, in go
      return once_fn(*arg, **kw)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 199, in first_connect
      dialect.initialize(c)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/dialects/mssql/base.py", line 2378, in initialize
      super(MSDialect, self).initialize(connection)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 318, in initialize
      self.returns_unicode_strings = self._check_unicode_returns(connection)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 415, in _check_unicode_returns
      results = {check_unicode(test) for test in tests}

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 415, in <setcomp>
      results = {check_unicode(test) for test in tests}

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 384, in check_unicode
      connection._cursor_execute(cursor, statement, parameters)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1321, in _cursor_execute
      e, statement, parameters, cursor, context

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
      util.raise_from_cause(sqlalchemy_exception, exc_info)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
      reraise(type(exception), exception, tb=exc_tb, cause=cause)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
      raise value.with_traceback(tb)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1318, in _cursor_execute
      self.dialect.do_execute(cursor, statement, parameters, context)

      File "/home/ops/.pyenv/versions/3.6.10/envs/airflow_env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
      cursor.execute(statement, parameters)

      sqlalchemy.exc.StatementError: (builtins.SystemError) <class 'pyodbc.Error'> returned a result with an error set

      [SQL: SELECT CAST('test unicode returns' AS NVARCHAR(60)) AS anon_1]

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              amith1893 Amith Gopal
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: