[AMBARI-22574] Failed to restart services on PPC cluster post Ambari upgrade for IOP/HDP migration

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.1
• Fix Version/s: 2.6.1
• Component/s: ambari-server
• Assignee: Di Li
• Reporter: Di Li
• Created: 01/Dec/17 18:56
• Updated: 05/Dec/17 19:46
• Resolved: 05/Dec/17 19:13

Description

The cluster started as IOP 4.2.5 on PPC with NameNode HA enabled. I upgraded Ambari to Ambari 2.6.1. The NN HA setting does not matter to the issue.

After the Ambari upgrade, I can't restart services by clicking the "Restart" button on each service with that "restart needed" icon. Nor can I abort the requests.

Ambari server log contains the following error on unable to find redhat-ppc7 os type.
java.lang.RuntimeException: org.apache.ambari.server.controller.spi.SystemException: Operating System matching redhat-ppc7 could not be found
at org.apache.ambari.server.actionmanager.ExecutionCommandWrapper.getExecutionCommand(ExecutionCommandWrapper.java:253)
at org.apache.ambari.server.actionmanager.ActionScheduler.abortOperationsForStage(ActionScheduler.java:880)
at org.apache.ambari.server.actionmanager.ActionScheduler.processCancelledRequestsList(ActionScheduler.java:1172)
at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:333)
at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:310)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ambari.server.controller.spi.SystemException: Operating System matching redhat-ppc7 could not be found
at org.apache.ambari.server.state.stack.upgrade.RepositoryVersionHelper.getOSEntityForHost(RepositoryVersionHelper.java:422)
at org.apache.ambari.server.actionmanager.ExecutionCommandWrapper.getExecutionCommand(ExecutionCommandWrapper.java:247)

repo_version table had "redhat7" as the os type in the repository information. I worked around the issue by updating repo_version table to have redhat-ppc7 as the os type via the following SQL query:

update repo_version set repositories='[{"OperatingSystems/ambari_managed_repositories":"true","repositories":[

{"Repositories/repo_id":"IOP-4.2.5","Repositories/base_url":"http://birepo-build.svl.ibm.com/repos/IOP/RHEL7/ppc64le/4.2.5.0/20170602_1749","Repositories/repo_name":"IOP"}

,

{"Repositories/repo_id":"IOP-UTILS-1.3","Repositories/base_url":"http://birepo-build.svl.ibm.com/repos/IOP-UTILS/RHEL7/ppc64le/1.3","Repositories/repo_name":"IOP-UTILS"}

],"OperatingSystems/os_type":"redhat-ppc7"}]' where repo_version_id = 1;

Issue Links

• is duplicated by: AMBARI-22593 Unable to perform ambari actions on IOP 4.2.5 post ambari upgrade on PPC
• links to: Review Board