[AMBARI-22864] Agent commands hang even after freeing up disk space on the host

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.2
• Fix Version/s: 2.6.2
• Component/s: ambari-agent, ambari-server
• Labels: pull-request-available system_test
• Assignee: Andrew Onischuk
• Reporter: Vivek Sharma
• Created: 29/Jan/18 11:33
• Updated: 01/Feb/18 10:57
• Resolved: 01/Feb/18 10:57

Description

STR

1. Install a cluster with Ambari-2.6.2 and HDP-2.6.4.0
2. Go the host (say host1) running Nimbus component and restart Nimbus
3. Fill up the disk space on host1 (in my test, the disk space was filled up on the host running Nimbus component)
4. Try to restart Nimbus. Nimbus restart expectedly fails with error:

   Caught an exception while executing custom service command: <type 'exceptions.IOError'>: [Errno 28] No space left on device; [Errno 28] No space left on device
   

5. Now free up the disk space on host1 and try to restart Nimbus

 

Result
Nimbus restart command hangs and eventually times out

Looks like the issue is because the action queue is unable to create new command for Nimbus restart.

Issue Links

• links to: GitHub Pull Request #240
• links to: GitHub Pull Request #241