Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.1.0
-
None
Description
1. Install cluster with Ambari 2.1 and HDP 2.3
2. Add services HDFS, YARN, MR, ZK, and HBaste
3. Perform several Stop All and Start All on HDFS service
4. Periodically, HBase Master will crash
This was a non-HA cluster.
2015-06-02 09:34:24,865 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException 2015-06-02 09:34:24,866 WARN [ip-172-31-33-225:16000.activeMasterManager] hdfs.DFSClient: DFS Read org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896) at java.io.DataInputStream.readFully(DataInputStream.java:195) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646) at java.lang.Thread.run(Thread.java:745) 2015-06-02 09:34:24,870 FATAL [ip-172-31-33-225:16000.activeMasterManager] master.HMaster: Failed to become active master org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-925466282-172.31.33.226-1433234647051:blk_1073741829_1005 file=/apps/hbase/data/hbase.id at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896) at java.io.DataInputStream.readFully(DataInputStream.java:195) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.hadoop.hbase.util.FSUtils.getClusterId(FSUtils.java:816) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:474) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:649) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646) at java.lang.Thread.run(Thread.java:745)
Attachments
Attachments
Issue Links
- links to