[AMBARI-11743] NameNode is forced to leave safemode, which causes HBMaster master to crash if done too quickly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.0, 2.0.3
• Component/s: ambari-server
• Assignee: Alejandro Fernandez
• Reporter: Alejandro Fernandez
• Created: 06/Jun/15 00:21
• Updated: 05/Aug/15 22:36
• Resolved: 09/Jun/15 01:51

Description

1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash

This was a non-HA cluster.

2015-06-02 09:34:24,865 WARN  [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2015-06-02 09:34:24,866 WARN  [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at java.io.DataInputStream.readFully(DataInputStream.java:169)
	at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
	at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
	at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
	at java.lang.Thread.run(Thread.java:745)
2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id
	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945)
	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604)
	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at java.io.DataInputStream.readFully(DataInputStream.java:169)
	at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816)
	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474)
	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649)
	at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
	at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: Code Review patch