Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-5036

Issue running resetdb with test config from code

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.4
    • Fix Version/s: None
    • Component/s: database
    • Labels:
      None

      Description

      In our local test of custom operators we've been doing an initdb and resetdb before each test. This worked fine in previous versions including 1.10.3. We recently tried the 1.10.4rc3 release and hit an issue.

      If I do an 'airflow initdb' and 'airflow resetdb' from command line it seems to work fine.

      If I run the following using Airflow 1.10.4rc3 on a clean db (ie: empty AIRFLOW_HOME).

       

      from airflow import configuration
      from airflow.utils import db
      configuration.load_test_config()
      db.initdb()
      db.resetdb(rbac=False)
      

       

      I get this error:

      INFO [alembic.runtime.migration] Context impl SQLiteImpl.
      INFO [alembic.runtime.migration] Will assume non-transactional DDL.
      INFO [alembic.runtime.migration] Context impl SQLiteImpl.
      INFO [alembic.runtime.migration] Will assume non-transactional DDL.
      INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema
      INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
      alembic/ddl/sqlite.py:39: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
      "Skipping unsupported ALTER for "
      INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
      INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
      INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
      Traceback (most recent call last):
      File "sqlalchemy/engine/base.py", line 1244, in _execute_context
      cursor, statement, parameters, context
      File "sqlalchemy/engine/default.py", line 552, in do_execute
      cursor.execute(statement, parameters)
      sqlite3.OperationalError: index idx_job_state_heartbeat already exists
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "airflow/utils/db.py", line 406, in resetdb
      initdb(rbac)
      File "airflow/utils/db.py", line 106, in initdb
      upgradedb()
      File "airflow/utils/db.py", line 377, in upgradedb
      command.upgrade(config, 'heads')
      File "alembic/command.py", line 276, in upgrade
      script.run_env()
      File "alembic/script/base.py", line 475, in run_env
      util.load_python_file(self.dir, "env.py")
      File "alembic/util/pyfiles.py", line 90, in load_python_file
      module = load_module_py(module_id, path)
      File "alembic/util/compat.py", line 177, in load_module_py
      spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "airflow/migrations/env.py", line 92, in <module>
      run_migrations_online()
      File "airflow/migrations/env.py", line 86, in run_migrations_online
      context.run_migrations()
      File "<string>", line 8, in run_migrations
      File "alembic/runtime/environment.py", line 839, in run_migrations
      self.get_context().run_migrations(**kw)
      File "alembic/runtime/migration.py", line 362, in run_migrations
      step.migration_fn(**kw)
      File "airflow/migrations/versions/52d714495f0_job_id_indices.py", line 38, in upgrade
      ['state', 'latest_heartbeat'], unique=False)
      File "<string>", line 8, in create_index
      File "<string>", line 3, in create_index
      File "alembic/operations/ops.py", line 975, in create_index
      return operations.invoke(op)
      File "alembic/operations/base.py", line 345, in invoke
      return fn(self, operation)
      File "alembic/operations/toimpl.py", line 88, in create_index
      operations.impl.create_index(idx)
      File "alembic/ddl/impl.py", line 276, in create_index
      self._exec(schema.CreateIndex(index))
      File "alembic/ddl/impl.py", line 134, in _exec
      return conn.execute(construct, *multiparams, **params)
      File "sqlalchemy/engine/base.py", line 988, in execute
      return meth(self, multiparams, params)
      File "sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
      return connection._execute_ddl(self, multiparams, params)
      File "sqlalchemy/engine/base.py", line 1050, in _execute_ddl
      compiled,
      File "sqlalchemy/engine/base.py", line 1248, in _execute_context
      e, statement, parameters, cursor, context
      File "sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
      util.raise_from_cause(sqlalchemy_exception, exc_info)
      File "sqlalchemy/util/compat.py", line 398, in raise_from_cause
      reraise(type(exception), exception, tb=exc_tb, cause=cause)
      File "sqlalchemy/util/compat.py", line 152, in reraise
      raise value.with_traceback(tb)
      File "sqlalchemy/engine/base.py", line 1244, in _execute_context
      cursor, statement, parameters, context
      File "sqlalchemy/engine/default.py", line 552, in do_execute
      cursor.execute(statement, parameters)
      sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) index idx_job_state_heartbeat already exists
      [SQL: CREATE INDEX idx_job_state_heartbeat ON job (state, latest_heartbeat)]
      (Background on this error at: http://sqlalche.me/e/e3q8)
      

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mejran Marcin Mejran
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: