[AMBARI-22216] Ambari Schema Upgrade Failed during Ambari Upgrade

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.0
• Component/s: ambari-server
• Assignee: Dmitry Lysnichenko
• Reporter: Dmitry Lysnichenko
• Created: 12/Oct/17 12:23
• Updated: 23/Oct/17 04:20
• Resolved: 23/Oct/17 04:20

Description

Ambari Schema Upgrade Failed, Upgrade from : 2510 to 2600 (Also seen from 2430 to 2600)

2017-10-03 06:30:19,761 INFO com.hw.ambari.ui.util.cluster_managers.CommandExecutor.executeCommandSequence(): Sending command [ambari-server upgrade --verbose]
2017-10-03 06:30:19,761 INFO com.hw.ambari.ui.util.cluster_managers.CommandExecutor.executeCommandSequence(): Sending command []
2017-10-03 06:30:19,861 DEBUG com.hw.ambari.ui.util.cluster_managers.ProcessData.buildOutputAndErrorStreamData(): stdin: is not a tty
2017-10-03 06:30:26,973 INFO com.hw.ambari.ui.util.cluster_managers.CommandExecutor.executeCommandSequence(): [OUTPUT STREAM]
Using python  /usr/bin/python
Upgrading ambari-server
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
INFO: Updating Ambari Server properties in ambari-env.sh ...
WARNING: Original file ambari-env.sh kept
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: No mpack replay logs found. Skipping replaying mpack commands
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Fixing database objects owner
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (y)? INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Upgrading database schema
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1
INFO:
process_pid=25159
INFO: Return code from schema upgrade command, retcode = 1
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Error output from schema upgrade command:
ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: Cannot drop index 'FK_servicecomponentdesiredstate_desired_stack_id': needed in a foreign key constraint
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
Caused by: java.sql.SQLException: Cannot drop index 'FK_servicecomponentdesiredstate_desired_stack_id': needed in a foreign key constraint
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974)
at org.apache.ambari.server.upgrade.UpgradeCatalog260.updateServiceComponentDesiredStateTable(UpgradeCatalog260.java:386)
at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:185)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more


ERROR: Ambari server upgrade failed. Please look at /var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed.

ambari-server.log

03 Oct 2017 06:30:26,349  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate ADD desired_repo_version_id BIGINT NOT NULL DEFAULT 1
03 Oct 2017 06:30:26,389  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate  MODIFY desired_repo_version_id BIGINT
03 Oct 2017 06:30:26,435  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate  MODIFY desired_repo_version_id BIGINT NOT NULL
03 Oct 2017 06:30:26,490  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate ADD repo_state VARCHAR(255) NOT NULL DEFAULT 'CURRENT'
03 Oct 2017 06:30:26,549  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate  MODIFY repo_state VARCHAR(255)
03 Oct 2017 06:30:26,598  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate  MODIFY repo_state VARCHAR(255) NOT NULL
03 Oct 2017 06:30:26,742  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate ADD CONSTRAINT FK_scds_desired_repo_id FOREIGN KEY (desired_repo_version_id) REFERENCES repo_version (repo_version_id)
03 Oct 2017 06:30:26,782  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate DROP FOREIGN KEY FK_scds_desired_stack_id
03 Oct 2017 06:30:26,802  INFO [main] DBAccessorImpl:874 - Executing query: ALTER TABLE servicecomponentdesiredstate DROP COLUMN desired_stack_id
03 Oct 2017 06:30:26,817 ERROR [main] DBAccessorImpl:880 - Error executing query: ALTER TABLE servicecomponentdesiredstate DROP COLUMN desired_stack_id
java.sql.SQLException: Cannot drop index 'FK_servicecomponentdesiredstate_desired_stack_id': needed in a foreign key constraint
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974)
at org.apache.ambari.server.upgrade.UpgradeCatalog260.updateServiceComponentDesiredStateTable(UpgradeCatalog260.java:386)
at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:185)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
03 Oct 2017 06:30:26,822 ERROR [main] SchemaUpgradeHelper:202 - Upgrade failed.

Issue Links

• links to: review_request_