[AMBARI-18422] Unable to cleanly delete host using ambari 2.2 rest API

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2-next
• Component/s: ambari-server, blueprints

Environment

Cluster with every host being a container

• Flags: Important
• Assignee: Unassigned
• Reporter: Kartik Mathur
• Created: 20/Sep/16 06:47
• Updated: 20/Sep/16 06:48

Description

I have a 7 node (7 containers) HDP 2.4 cluster with ambari 2.2

I am trying to delete a host using the following steps -

I am using ambari rest apis-

1) Decommission data node and node manager on the host
2) Stop all services running on the host
3) Delete all services on the host
uptill this point everything is fine
4) Delete host using the api
5) stop ambari-agent on the deleted host.

The problem is in step 4 , which always gives me "server error" while trying to delete and the ambari logs gives "No Host" exception.

Interestingly the host disappears from Ambari UI however the db is in an inconsistent state and there are alerts complaining for dead data nodes.

Even tried the same from Ambari UI but I get the same error.