[AMBARI-22257] Metrics collector fails to stop after Datanode is stopped in distributed mode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.6.0
• Component/s: ambari-metrics
• Assignee: Siddharth Wagle
• Reporter: Siddharth Wagle
• Created: 17/Oct/17 17:22
• Updated: 17/Oct/17 18:54
• Resolved: 17/Oct/17 17:57

Description

AMS collector stop failed due to timeout at the ams-hbase regionserver stop. The log contains lots of exceptions related to DN connection issues during the stop. The problem here is that DNs were stopped before the collector.

2017-10-17 14:29:10,689 ERROR [Thread-274] hdfs.DFSClient: Failed to close inode 17762
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/ams/hbase/WALs/ctr-e134-1499953498516-230429-01-000007.hwx.site,61320,1508248489809/ctr-e134-1499953498516-230429-01-000007.hwx.site%2C61320%2C1508248489809.default.1508250548392 could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1719)