Details
Description
We have InitializationMonitor thread in HMaster which detects Zombie Hmaster based on hbase.master.initializationmonitor.timeout _and halts if _hbase.master.initializationmonitor.haltontimeout set true.
After HBASE-19694, HMaster initialization order was correted. Hmaster is set active after Initializing ZK system trackers as follows,
status.setStatus("Initializing ZK system trackers"); initializeZKBasedSystemTrackers(); status.setStatus("Loading last flushed sequence id of regions"); try { this.serverManager.loadLastFlushedSequenceIds(); } catch (IOException e) { LOG.debug("Failed to load last flushed sequence id of regions" + " from file system", e); } // Set ourselves as active Master now our claim has succeeded up in zk. this.activeMaster = true;
But Zombie detector thread is started at the begining phase of finishActiveMasterInitialization(),
private void finishActiveMasterInitialization(MonitoredTask status) throws IOException, InterruptedException, KeeperException, ReplicationException { Thread zombieDetector = new Thread(new InitializationMonitor(this), "ActiveMasterInitializationMonitor-" + System.currentTimeMillis()); zombieDetector.setDaemon(true); zombieDetector.start();
During zombieDetector execution "master.isActiveMaster()" will be false, so it won't wait and cant detect zombie master.
@Override public void run() { try { while (!master.isStopped() && master.isActiveMaster()) { Thread.sleep(timeout); if (master.isInitialized()) { LOG.debug("Initialization completed within allotted tolerance. Monitor exiting."); } else { LOG.error("Master failed to complete initialization after " + timeout + "ms. Please" + " consider submitting a bug report including a thread dump of this process."); if (haltOnTimeout) { LOG.error("Zombie Master exiting. Thread dump to stdout"); Threads.printThreadInfo(System.out, "Zombie HMaster"); System.exit(-1); } } } } catch (InterruptedException ie) { LOG.trace("InitMonitor thread interrupted. Existing."); } } }
Attachments
Attachments
Issue Links
- relates to
-
HBASE-19694 The initialization order for a fresh cluster is incorrect
- Resolved