Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-1841

Sqoop2: Upgrade from 1.99.1 to 1.99.4 onwards is broken

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.99.4
    • 1.99.6
    • sqoop2-derby-repository
    • None

    Description

      A constraint on the SQ_JOB table was added in 1.99.3 that doesn't have corresponding upgrade code.

      7:20:49.070 AM	INFO	org.apache.sqoop.server.ServerInitializer	
Sqoop server has been correctly terminated
7:57:52.844 AM	INFO	org.apache.sqoop.core.PropertiesConfigurationProvider	
Starting config file poller thread
7:57:52.847 AM	INFO	org.apache.sqoop.tools.tool.UpgradeTool	
Initializing the RepositoryManager with immutable option turned off.
7:57:52.853 AM	INFO	org.apache.sqoop.repository.RepositoryManager	
Setting system properties: {derby.stream.error.file=/var/log/sqoop2/derbyrepo.log}
7:57:52.888 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryContext	
[repo-ctx] handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler, conn-url=jdbc:derby:/var/lib/sqoop2/repository/db;create=true, driver=org.apache.derby.jdbc.EmbeddedDriver, user=null, password=*****, jdbc-props={}, tx-isolation=READ_COMMITTED, max-conn=10
7:57:53.836 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
DerbyRepositoryHandler initialized.
7:57:53.843 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryProvider	
JdbcRepositoryProvider initialized
7:57:53.843 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryProvider	
JdbcRepository initialized.
7:57:53.843 AM	INFO	org.apache.sqoop.repository.RepositoryManager	
Creating or updating respository at bootup
7:57:54.491 AM	INFO	org.apache.sqoop.repository.JdbcRepository	
Creating repository schema objects
7:57:55.478 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(CREATE TABLE SQOOP.SQ_SYSTEM (SQM_ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, SQM_KEY VARCHAR(64), SQM_VALUE VARCHAR(64) )) Update count: 0
7:57:55.605 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_CONNECTION ADD SQN_ENABLED BOOLEAN DEFAULT TRUE) Update count: 0
7:57:55.658 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_JOB ADD SQB_ENABLED BOOLEAN DEFAULT TRUE) Update count: 0
7:57:55.680 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_CONNECTION ADD SQN_CREATION_USER VARCHAR(32) DEFAULT NULL) Update count: 0
7:57:55.703 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_CONNECTION ADD SQN_UPDATE_USER VARCHAR(32) DEFAULT NULL) Update count: 0
7:57:55.721 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_JOB ADD SQB_CREATION_USER VARCHAR(32) DEFAULT NULL) Update count: 0
7:57:55.738 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_JOB ADD SQB_UPDATE_USER VARCHAR(32) DEFAULT NULL) Update count: 0
7:57:55.760 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_SUBMISSION ADD SQS_CREATION_USER VARCHAR(32) DEFAULT NULL) Update count: 0
7:57:55.777 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_SUBMISSION ADD SQS_UPDATE_USER VARCHAR(32) DEFAULT NULL) Update count: 0
7:57:55.800 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_SUBMISSION ALTER COLUMN SQS_EXTERNAL_ID SET DATA TYPE VARCHAR(50)) Update count: 0
7:57:55.814 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_CONNECTOR ALTER COLUMN SQC_VERSION SET DATA TYPE VARCHAR(64)) Update count: 0
7:57:55.862 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(CREATE TABLE SQOOP.SQ_DIRECTION (SQD_ID BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, SQD_NAME VARCHAR(64))) Update count: 0
7:57:55.893 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(RENAME COLUMN SQOOP.SQ_FORM.SQF_OPERATION TO SQF_DIRECTION) Update count: 0
7:57:55.919 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(RENAME COLUMN SQOOP.SQ_JOB.SQB_CONNECTION TO SQB_FROM_CONNECTION) Update count: 0
7:57:55.935 AM	INFO	org.apache.sqoop.repository.derby.DerbyRepositoryHandler	
QUERY(ALTER TABLE SQOOP.SQ_JOB ADD COLUMN SQB_TO_CONNECTION BIGINT) Update count: 0
7:57:55.954 AM	INFO	org.apache.sqoop.repository.JdbcRepositoryTransaction	
Attempting transaction commit
7:57:55.958 AM	ERROR	org.apache.sqoop.tools.tool.UpgradeTool	
Can't finish upgrading RepositoryManager, Driver and ConnectionManager:
org.apache.sqoop.common.SqoopException: DERBYREPO_0003:Unable to run specified query - ALTER TABLE SQOOP.SQ_JOB DROP CONSTRAINT SQOOP.FK_SQB_SQN
	at org.apache.sqoop.repository.derby.DerbyRepositoryHandler.runQuery(DerbyRepositoryHandler.java:2633)
	at org.apache.sqoop.repository.derby.DerbyRepositoryHandler.createOrUpgradeRepository(DerbyRepositoryHandler.java:424)
	at org.apache.sqoop.repository.JdbcRepository$1.doIt(JdbcRepository.java:131)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:90)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)
	at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:127)
	at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:123)
	at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
	at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
	at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.sqoop.tomcat.TomcatToolRunner.main(TomcatToolRunner.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.catalina.startup.Tool.main(Tool.java:225)
Caused by: org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from pool failed
	at org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:113)
	at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
	at org.apache.sqoop.repository.derby.DerbyRepositoryHandler.runQuery(DerbyRepositoryHandler.java:2609)
	... 19 more
Caused by: java.sql.SQLSyntaxErrorException: ALTER TABLE failed. There is no constraint 'SQOOP.FK_SQB_SQN' on table '"SQOOP"."SQ_JOB"'. 
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown Source)
	at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
	at org.apache.commons.dbcp.PoolingConnection.makeObject(PoolingConnection.java:285)
	at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
	at org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:107)
	... 22 more
Caused by: java.sql.SQLException: ALTER TABLE failed. There is no constraint 'SQOOP.FK_SQB_SQN' on table '"SQOOP"."SQ_JOB"'. 
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
	... 38 more
Caused by: ERROR 42X86: ALTER TABLE failed. There is no constraint 'SQOOP.FK_SQB_SQN' on table '"SQOOP"."SQ_JOB"'. 
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.impl.sql.compile.TableElementList.validate(Unknown Source)
	at org.apache.derby.impl.sql.compile.AlterTableNode.bindStatement(Unknown Source)
	at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
	at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
	at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
	... 32 more
7:57:55.966 AM	INFO	org.apache.sqoop.core.PropertiesConfigurationProvider	
Shutting down configuration poller thread

      Attachments

        1. SQOOP-1841.1.patch
          16 kB
          Abraham Elmahrek
        2. SQOOP-1841.0.patch
          13 kB
          Abraham Elmahrek

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SQOOP-1790 Sqoop2: Upgrade configs with name conflicts

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SQOOP-2129 Sqoop2: SQ_SUBMISSION deletes should cascade

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link RB

          Activity

            People

              abec Abraham Elmahrek
              abec Abraham Elmahrek
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: