Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.99.0
-
None
-
None
-
Reviewed
Description
Playing w/ MTTR recovery on trunk, master starting up deadlocked:
Waiting to finish active master initialization:
"ActiveMasterManager" daemon prio=10 tid=0x00007fafb5dc3800 nid=0x5fb5 waiting for monitor entry [0x00007faf8f57d000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683) - waiting to lock <0x000000064ab4b9a8> (a java.lang.Object) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830) at org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77) at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:164) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:107) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539) at org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140) at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164) at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242) at java.lang.Thread.run(Thread.java:744)
... but the master servlet has the lock while trying to access master:
"686004346@qtp-2101021459-0" daemon prio=10 tid=0x00007fafb5d2a800 nid=0x5fb1 waiting on condition [0x00007faf8f87f000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562) - locked <0x000000064ab4b9a8> (a java.lang.Object) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805) - locked <0x000000064ab4b9a8> (a java.lang.Object) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481) at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321) at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530) at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255) at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382) at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372) at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102) ...