HBase
  1. HBase
  2. HBASE-10897

On master start, deadlock if refresh UI

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.99.0
    • Fix Version/s: 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Playing w/ MTTR recovery on trunk, master starting up deadlocked:

      Waiting to finish active master initialization:

      "ActiveMasterManager" daemon prio=10 tid=0x00007fafb5dc3800 nid=0x5fb5 waiting for monitor entry [0x00007faf8f57d000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1683)
              - waiting to lock <0x000000064ab4b9a8> (a java.lang.Object)
              at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:53)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1029)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:989)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:830)
              at org.apache.hadoop.hbase.client.ConnectionAdapter.getRegionLocation(ConnectionAdapter.java:305)
              at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:77)
              at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:118)
              at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
              at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:264)
              at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:169)
              at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:164)
              at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:107)
              at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:766)
              at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:539)
              at org.apache.hadoop.hbase.catalog.MetaReader.fullScanOfMeta(MetaReader.java:140)
              at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.isMetaTableUpdated(MetaMigrationConvertingToPB.java:164)
              at org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB.updateMetaIfNecessary(MetaMigrationConvertingToPB.java:131)
              at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:567)
              at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:147)
              at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1242)
              at java.lang.Thread.run(Thread.java:744)
      
      

      ... but the master servlet has the lock while trying to access master:

      "686004346@qtp-2101021459-0" daemon prio=10 tid=0x00007fafb5d2a800 nid=0x5fb1 waiting on condition [0x00007faf8f87f000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1562)
              - locked <0x000000064ab4b9a8> (a java.lang.Object)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1597)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1805)
              - locked <0x000000064ab4b9a8> (a java.lang.Object)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.listTables(ConnectionManager.java:2481)
              at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:321)
              at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__userTables(MasterStatusTmplImpl.java:530)
              at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:255)
              at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:382)
              at org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:372)
              at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:102)
      ...
      
      1. hbase-10897.patch
        0.7 kB
        Jimmy Xiang
      2. hbase-10897_v2.patch
        7 kB
        Jimmy Xiang
      3. hbase-10897_v3.patch
        9 kB
        Jimmy Xiang
      4. hbase-10897_v4.patch
        10 kB
        Jimmy Xiang

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development