Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-1235

Odd behaviour when all gunicorn workers die

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 2.0.0
    • Component/s: webserver
    • Labels:
      None

      Description

      The webserver has sometimes stopped responding to port 443, and today I found the issue - I had a misconfigured resolv.conf that made it unable to talk to my postgresql. This was the root cause, but the way airflow webserver behaved was a bit odd.

      It seems that when all gunicorn workers failed to start, the gunicorn master shut down. However, the main process (the one that starts gunicorn master) did not shut down, so there was no way of detecting the failed status of webserver from e.g. systemd or init script.

      Full traceback leading to stale webserver process:

      May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794] [ERROR] Exception in worker process:
      May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1122, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, self._timeout)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py", line 145, in get
      May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
      May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
      May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above exception, another exception occurred:
      May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
      May 21 09:51:57 airmaster01 airflow[26451]: return fn()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 387, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 766, in _checkout
      May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 516, in checkout
      May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
      May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
      May 21 09:51:57 airmaster01 airflow[26451]: raise value
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
      May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 461, in __init__
      May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 651, in __connect
      May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 393, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
      May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could not translate host name "airstorage01.example.com" to address: Name or service not known
      May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct cause of the following exception:
      May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
      May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py", line 118, in init_process
      May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py", line 67, in wsgi
      May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
      May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
      May 21 09:51:57 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line 366, in import_app
      May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
      May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line 161, in cached_app
      May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line 60, in create_app
      May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", line 1975, in <module>
      May 21 09:51:57 airmaster01 airflow[26451]: class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", line 2057, in ChartModelView
      May 21 09:51:57 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
      May 21 09:51:57 airmaster01 airflow[26451]: return self._execute_and_instances(context)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2876, in _execute_and_instances
      May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2885, in _get_bind_args
      May 21 09:51:57 airmaster01 airflow[26451]: **kw
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2867, in _connection_from_session
      May 21 09:51:57 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 966, in connection
      May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified.
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 971, in _connection_for_bind
      May 21 09:51:57 airmaster01 airflow[26451]: closed.  This flag only has an effect if this :class:`.Session` is
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind
      May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2112, in contextual_connect
      May 21 09:51:57 airmaster01 airflow[26451]: self._wrap_pool_connect(self.pool.connect, None),
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
      May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
      May 21 09:51:57 airmaster01 airflow[26451]: exc_info
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
      May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
      May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
      May 21 09:51:57 airmaster01 airflow[26451]: return fn()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 387, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 766, in _checkout
      May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 516, in checkout
      May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
      May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
      May 21 09:51:57 airmaster01 airflow[26451]: raise value
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
      May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 461, in __init__
      May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 651, in __connect
      May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 393, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
      May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "airstorage01.example.com" to address: Name or service not known
      May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1122, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, self._timeout)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py", line 145, in get
      May 21 09:51:57 airmaster01 airflow[26451]: raise Empty
      May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty
      May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above exception, another exception occurred:
      May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
      May 21 09:51:57 airmaster01 airflow[26451]: return fn()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 387, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 766, in _checkout
      May 21 09:51:57 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 516, in checkout
      May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
      May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
      May 21 09:51:57 airmaster01 airflow[26451]: raise value
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
      May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
      May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 461, in __init__
      May 21 09:51:57 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 651, in __connect
      May 21 09:51:57 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 393, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
      May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
      May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could not translate host name "airstorage01.example.com" to address: Name or service not known
      May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the direct cause of the following exception:
      May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last):
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
      May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py", line 118, in init_process
      May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi()
      May 21 09:51:57 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py", line 67, in wsgi
      May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load()
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
      May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp()
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
      May 21 09:51:58 airmaster01 airflow[26451]: return util.import_app(self.app_uri)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line 366, in import_app
      May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__)
      May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in <module>
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line 161, in cached_app
      May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", line 60, in create_app
      May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", line 1975, in <module>
      May 21 09:51:58 airmaster01 airflow[26451]: class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", line 2057, in ChartModelView
      May 21 09:51:58 airmaster01 airflow[26451]: .group_by(models.Connection.conn_id)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
      May 21 09:51:58 airmaster01 airflow[26451]: return self._execute_and_instances(context)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2876, in _execute_and_instances
      May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2885, in _get_bind_args
      May 21 09:51:58 airmaster01 airflow[26451]: **kw
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2867, in _connection_from_session
      May 21 09:51:58 airmaster01 airflow[26451]: conn = self.session.connection(**kw)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 966, in connection
      May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified.
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 971, in _connection_for_bind
      May 21 09:51:58 airmaster01 airflow[26451]: closed.  This flag only has an effect if this :class:`.Session` is
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind
      May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect()
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2112, in contextual_connect
      May 21 09:51:58 airmaster01 airflow[26451]: self._wrap_pool_connect(self.pool.connect, None),
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
      May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
      May 21 09:51:58 airmaster01 airflow[26451]: exc_info
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
      May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
      May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
      May 21 09:51:58 airmaster01 airflow[26451]: return fn()
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 387, in connect
      May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionFairy._checkout(self)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 766, in _checkout
      May 21 09:51:58 airmaster01 airflow[26451]: fairy = _ConnectionRecord.checkout(pool)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 516, in checkout
      May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get()
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
      May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow()
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
      May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type, exc_value, exc_tb)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
      May 21 09:51:58 airmaster01 airflow[26451]: raise value
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
      May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection()
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
      May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 461, in __init__
      May 21 09:51:58 airmaster01 airflow[26451]: self.__connect(first_connect_check=True)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", line 651, in __connect
      May 21 09:51:58 airmaster01 airflow[26451]: connection = pool._invoke_creator(self)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
      May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs, **cparams)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 393, in connect
      May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, **cparams)
      May 21 09:51:58 airmaster01 airflow[26451]: File "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
      May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
      May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "airstorage01.example.com" to address: Name or service not known
      May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] [23794] [INFO] Worker exiting (pid: 23794)
      May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
      May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
      May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23561] [INFO] Worker exiting (pid: 23561)
      May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
      May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
      May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23759] [INFO] Worker exiting (pid: 23759)
      May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
      May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
      May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23620] [INFO] Worker exiting (pid: 23620)
      May 21 09:51:58 airmaster01 airflow[26451]: /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
      May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), ExtDeprecationWarning
      May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] [23689] [INFO] Worker exiting (pid: 23689)
      May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464] [INFO] Shutting down: Master
      May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] [26464] [INFO] Reason: Worker failed to boot.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                forsberg Erik Forsberg
              • Votes:
                5 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: