Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1841

Sqoop2: Upgrade from 1.99.1 to 1.99.4 onwards is broken

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.4
    • Fix Version/s: 1.99.6
    • Labels:
      None

      Description

      A constraint on the SQ_JOB table was added in 1.99.3 that doesn't have corresponding upgrade code.

      7:20:49.070 AM	INFO	org.apache.sqoop.server.ServerInitializer	
      Sqoop server has been correctly terminated
      7:57:52.844 AM	INFO	org.apache.sqoop.core.PropertiesConfigurationProvider	
      Starting config file poller thread
      7:57:52.847 AM	INFO	org.apache.sqoop.tools.tool.UpgradeTool	
      Initializing the RepositoryManager with immutable option turned off.
      7:57:52.853 AM	INFO	org.apache.sqoop.repository.RepositoryManager	
      Setting system properties: {derby.stream.error.file=/var/log/sqoop2/derbyrepo.log}
      7:57:52.888 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryContext	
      [repo-ctx] handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler, conn-url=jdbc:derby:/var/lib/sqoop2/repository/db;create=true, driver=org.apache.derby.jdbc.EmbeddedDriver, user=null, password=*****, jdbc-props={}, tx-isolation=READ_COMMITTED, max-conn=10
      7:57:53.836 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      DerbyRepositoryHandler initialized.
      7:57:53.843 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryProvider	
      JdbcRepositoryProvider initialized
      7:57:53.843 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryProvider	
      JdbcRepository initialized.
      7:57:53.843 AM	INFO	org.apache.sqoop.repository.RepositoryManager	
      Creating or updating respository at bootup
      7:57:54.491 AM	INFO	org.apache.sqoop.repository.JdbcRepository	
      Creating repository schema objects
      7:57:55.478 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(CREATE TABLE SQOOP.SQ_SYSTEM (SQM_ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, SQM_KEY VARCHAR(64), SQM_VALUE VARCHAR(64) )) Update count: 0
      7:57:55.605 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_CONNECTION ADD SQN_ENABLED BOOLEAN DEFAULT TRUE) Update count: 0
      7:57:55.658 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_JOB ADD SQB_ENABLED BOOLEAN DEFAULT TRUE) Update count: 0
      7:57:55.680 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_CONNECTION ADD SQN_CREATION_USER VARCHAR(32) DEFAULT NULL) Update count: 0
      7:57:55.703 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_CONNECTION ADD SQN_UPDATE_USER VARCHAR(32) DEFAULT NULL) Update count: 0
      7:57:55.721 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_JOB ADD SQB_CREATION_USER VARCHAR(32) DEFAULT NULL) Update count: 0
      7:57:55.738 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_JOB ADD SQB_UPDATE_USER VARCHAR(32) DEFAULT NULL) Update count: 0
      7:57:55.760 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_SUBMISSION ADD SQS_CREATION_USER VARCHAR(32) DEFAULT NULL) Update count: 0
      7:57:55.777 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_SUBMISSION ADD SQS_UPDATE_USER VARCHAR(32) DEFAULT NULL) Update count: 0
      7:57:55.800 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_SUBMISSION ALTER COLUMN SQS_EXTERNAL_ID SET DATA TYPE VARCHAR(50)) Update count: 0
      7:57:55.814 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_CONNECTOR ALTER COLUMN SQC_VERSION SET DATA TYPE VARCHAR(64)) Update count: 0
      7:57:55.862 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(CREATE TABLE SQOOP.SQ_DIRECTION (SQD_ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, SQD_NAME VARCHAR(64))) Update count: 0
      7:57:55.893 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(RENAME COLUMN SQOOP.SQ_FORM.SQF_OPERATION TO SQF_DIRECTION) Update count: 0
      7:57:55.919 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(RENAME COLUMN SQOOP.SQ_JOB.SQB_CONNECTION TO SQB_FROM_CONNECTION) Update count: 0
      7:57:55.935 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
      QUERY(ALTER TABLE SQOOP.SQ_JOB ADD COLUMN SQB_TO_CONNECTION BIGINT) Update count: 0
      7:57:55.954 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryTransaction	
      Attempting transaction commit
      7:57:55.958 AM	ERROR	org.apache.sqoop.tools.tool.UpgradeTool	
      Can't finish upgrading RepositoryManager, Driver and ConnectionManager:
      org.apache.sqoop.common.SqoopException: DERBYREPO_0003:Unable to run specified query - ALTER TABLE SQOOP.SQ_JOB DROP CONSTRAINT SQOOP.FK_SQB_SQN
      	at org.apache.sqoop.repository.derby.DerbyRepositoryHandler.runQuery(DerbyRepositoryHandler.java:2633)
      	at org.apache.sqoop.repository.derby.DerbyRepositoryHandler.createOrUpgradeRepository(DerbyRepositoryHandler.java:424)
      	at org.apache.sqoop.repository.JdbcRepository$1.doIt(JdbcRepository.java:131)
      	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:90)
      	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
      	at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)
      	at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
      	at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
      	at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
      	at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.catalina.startup.Tool.main(Tool.java:225)
      Caused by: org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from pool failed
      	at org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:113)
      	at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
      	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
      	at org.apache.sqoop.repository.derby.DerbyRepositoryHandler.runQuery(DerbyRepositoryHandler.java:2609)
      	... 19 more
      Caused by: java.sql.SQLSyntaxErrorException: ALTER TABLE failed. There is no constraint 'SQOOP.FK_SQB_SQN' on table '"SQOOP"."SQ_JOB"'. 
      	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown Source)
      	at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
      	at org.apache.commons.dbcp.PoolingConnection.makeObject(PoolingConnection.java:285)
      	at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
      	at org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:107)
      	... 22 more
      Caused by: java.sql.SQLException: ALTER TABLE failed. There is no constraint 'SQOOP.FK_SQB_SQN' on table '"SQOOP"."SQ_JOB"'. 
      	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
      	... 38 more
      Caused by: ERROR 42X86: ALTER TABLE failed. There is no constraint 'SQOOP.FK_SQB_SQN' on table '"SQOOP"."SQ_JOB"'. 
      	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      	at org.apache.derby.impl.sql.compile.TableElementList.validate(Unknown Source)
      	at org.apache.derby.impl.sql.compile.AlterTableNode.bindStatement(Unknown Source)
      	at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
      	at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
      	at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
      	... 32 more
      7:57:55.966 AM	INFO	org.apache.sqoop.core.PropertiesConfigurationProvider	
      Shutting down configuration poller thread
      

        Attachments

        1. SQOOP-1841.0.patch
          13 kB
          Abraham Elmahrek
        2. SQOOP-1841.1.patch
          16 kB
          Abraham Elmahrek

          Issue Links

            Activity

              People

              • Assignee:
                abec Abraham Elmahrek
                Reporter:
                abec Abraham Elmahrek
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: