Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6844

Airflow scheduler explodes with RecursionError: maximum recursion depth exceeded

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.7
    • Fix Version/s: None
    • Component/s: DAG, scheduler
    • Labels:
    • Environment:
      OS X, SQLAlchemy -> MySQL, Reddis.
    • Flags:
      Important

      Description

      In Apache-AirFlow, I have written custom BaseOperator and executing that in a DAG, following is the code,

      Operator Code :

      from airflow.plugins_manager import AirflowPlugin from airflow.utils.decorators import apply_defaults

      class TestOperator(BaseOperator): template_fields = ('param1') ui_color = '#A7E6A7'

       

      {{@apply_defaults
      def _init_(self,param1,*args, **kwargs):
      self.param1 = param1
      super(TestOperator, self)._init_(*args, **kwargs)

      def execute(self):
      print ('welcome to airflow')}}

      class TestOperatorPlugin(AirflowPlugin): name = "TestOperator_plugin" operators = [TestOperator]

      DAG Code :

      from TestOperator import TestOperator from airflow import DAG from datetime import datetime

      prog_args = { 'depends_on_past': False, 'param1' : 'testOne' }

      testMYDAG = DAG('TestMYDAG', start_date = datetime(2020, 2, 18) , description='TestMYDAG', default_args = prog_args, schedule_interval=None)

      testOp = TestOperator(task_id='test_dag', dag=testMYDAG )

      testOp


      Once I start AirFlow webserver and scheduler, I am triggering DAG "TestMYDAG" manually. I get the following error and after this error console I don't get back till I delete all backend table entries for job, tasks in MySQL manually. Here is the error on Console ,

      UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functionsrule.endpoint File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner return self._run_view(f, args, **kwargs) File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2280, in index auto_complete_data=auto_complete_data) File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 388, in render return super(AirflowViewMixin, self).render(template, **kwargs) File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 308, in render return render_template(template, **kwargs) File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template ctx.app, File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render rv = template.render(context) File "/usr/local/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dags.html", line 20, in top-level template code {% extends "airflow/master.html" %} File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1005, in render return concat(self.root_render_func(self.new_context(vars))) File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dags.html", line 16, in root specific language governing permissions and limitations File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/master.html", line 16, in root specific language governing permissions and limitations File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/admin/master.html", line 16, in root specific language governing permissions and limitations File "/usr/local/lib/python3.7/site-packages/flask_admin/templates/bootstrap3/admin/base.html", line 26, in root File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/admin/master.html", line 90, in block_page_body function convertSecsToHumanReadable(seconds) { File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dags.html", line 79, in block_body File "/usr/local/lib/python3.7/site-packages/jinja2/runtime.py", line 262, in call return *obj(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", line 1607, in get_last_dagrun include_externally_triggered=include_externally_triggered) File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", line 81, in get_last_dagrun return query.first() File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3287, in first ret = list(self[0:1]) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3065, in __getitem return list(res) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3389, in iter return self._execute_and_instances(context) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3414, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute return meth(self, multiparams, params) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement distilled_params, File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context e, statement, parameters, cursor, context File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1478, in _handle_dbapi_exception util.reraise(*exc_info) File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise raise value File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context cursor, statement, parameters, context File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute cursor.execute(statement, parameters) File "/usr/local/lib/python3.7/site-packages/mysql/connector/cursor_cext.py", line 272, in execute self._handle_result(result) File "/usr/local/lib/python3.7/site-packages/mysql/connector/cursor_cext.py", line 163, in _handle_result self._handle_resultset() File "/usr/local/lib/python3.7/site-packages/mysql/connector/cursor_cext.py", line 651, in _handle_resultset self._rows = self._cnx.get_rows()[0] File "/usr/local/lib/python3.7/site-packages/mysql/connector/connection_cext.py", line 318, in get_rows else self._cmysql.fetch_row() SystemError: returned a result with an error set


      if scheduler is still running and not stopped after some time I am getting following error.

      ..... File "/usr/local/lib/python3.7/site-packages/airflow/utils/log/logging_mixin.py", line 112, in _propagate_log self.logger.log(self.level, remove_escape_codes(message)) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/init.py", line 1444, in log self._log(level, msg, args, *kwargs) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/*init.py", line 1514, in _log self.handle(record) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/init.py", line 1524, in handle self.callHandlers(record) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/init.py", line 1586, in callHandlers hdlr.handle(record) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/init.py", line 894, in handle self.emit(record) File "/usr/local/lib/python3.7/site-packages/airflow/utils/log/file_processor_handler.py", line 76, in emit self.handler.emit(record) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/init.py", line 1127, in emit StreamHandler.emit(self, record) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/init.py", line 1025, in emit msg = self.format(record) File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/init.py", line 869, in format return fmt.format(record) RecursionError: maximum recursion depth exceeded

      Please let me know if any questions, need more clarity.

      Thanks and Regards, Shubhada

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              shubhada317 Shubhada Anand
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: