Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-622

airflow cannot recover itself after mysql connection was once gone away and not trigger an error from email alert

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.6.2
    • Fix Version/s: None
    • Component/s: database
    • Labels:
    • Environment:
      airflow 1.6.1
      Ubuntu 14.04

      Description

      This issue occurs once the connections with airflow mysql gone away, even it recovered soon, but airflow hang there and print the following errors.
      Base on the suggestion of mysql, Session rollback method is needed.

      Also not alert mail for this case, and no retry, just silently wrap the log files.

      2016-11-10 16:04:30,361 [ERROR] file:jobs.py Line:582 - This Session's transaction has been rolled back by a nested rollback() call. To begin a new transaction, issue Session.rollback() first.
      Traceback (most recent call last):
      File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 580, in _execute
      self.prioritize_queued(executor=executor, dagbag=dagbag)
      File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/utils.py", line 122, in wrapper
      result = func(*args, **kwargs)
      File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 482, in prioritize_queued
      pools =

      {p.pool: p for p in session.query(models.Pool).all()}

      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2584, in all
      return list(self)
      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2732, in _iter_
      return self._execute_and_instances(context)
      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2745, in _execute_and_instances
      close_with_result=True)
      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2736, in _connection_from_session
      **kw)
      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 905, in connection
      execution_options=execution_options)
      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
      engine, execution_options)
      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
      self._assert_active()
      File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 218, in _assert_active
      "This Session's transaction has been rolled back "
      InvalidRequestError: This Session's transaction has been rolled back by a nested rollback() call. To begin a new transaction, issue Session.rollback() first.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hahanizhu SHENG JIANG TAO
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: