[AMBARI-22421] Ambari upgrade is not idempotent if it fails with service not found error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.2
• Component/s: ambari-server
• Assignee: Dmitry Lysnichenko
• Reporter: amarnath reddy pappu
• Created: 10/Nov/17 21:48
• Updated: 12/Apr/18 01:26
• Resolved: 27/Feb/18 12:51

Description

If Upgrade is failed with below exception and try to upgrade from that point after correcting the service not found issue.

06 Nov 2017 04:31:26,823 ERROR [main] SchemaUpgradeHelper:233 - Upgrade failed. 
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=XXXX, serviceName=NIFI
	at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
	at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
	at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
	at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:938)
	at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:231)
	at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:424)
06 Nov 2017 04:31:26,826 ERROR [main] SchemaUpgradeHelper:437 - Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException: Service not found, clusterName=usldwdevhdp, serviceName=NIFI
	at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:234)
	at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:424)
Caused by: org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=usldwdevhdp, serviceName=NIFI
	at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
	at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
	at org.apache.ambari.server.upgrade.UpdateAlertScriptPaths.executeDMLUpdates(UpdateAlertScriptPaths.java:46)
	at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:938)
	at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:231)

Upgrade fails with below exception

INFO: Updating Ambari Server properties in ambari.properties ...
WARNING: Can not find ambari.properties.rpmsave file from previous version, skipping import of settings
INFO: Updating Ambari Server properties in ambari-env.sh ...
INFO: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings. ambari-env.sh may not include any user customization.
INFO: Fixing database objects owner
Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (y)?
INFO: Upgrading database schema
INFO: Return code from schema upgrade command, retcode = 1
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Error output from schema upgrade command:
ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: cluster_version table does not contain repo_version_id column
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
Caused by: java.lang.IllegalArgumentException: cluster_version table does not contain repo_version_id column
        at org.apache.ambari.server.orm.DBAccessorImpl.getIntColumnValues(DBAccessorImpl.java:1536)
        at org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:507)
        at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:194)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
        ... 1 more


ERROR: Ambari server upgrade failed. Please look at /var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed.