Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19639

ITBLL can't go big because RegionTooBusyException... Above memstore limit

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Running ITBLLs, the basic link generator keeps failing because I run into exceptions like below:

      2017-12-26 19:23:45,284 INFO [main] org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator: Persisting current.length=1000000, count=1000000, id=Job: job_1513025868268_0062 Task: attempt_1513025868268_0062_m_000006_2, current=\x8B\xDB25\xA7*\x9A\xF5\xDEx\x83\xDF\xDC?\x94\x92, i=1000000
      2017-12-26 19:24:18,982 INFO [htable-pool3-t6] org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, table=IntegrationTestBigLinkedList, attempt=10/11 failed=524ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b., server=ve0538.halxg.cloudera.com,16020,1514343549993, memstoreSize=538084641, blockingMemStoreSize=536870912
      	at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
      	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
      	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
       on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, retrying after=10050ms, replay=524ops
      2017-12-26 19:24:29,061 INFO [htable-pool3-t6] org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, table=IntegrationTestBigLinkedList, attempt=11/11 failed=524ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b., server=ve0538.halxg.cloudera.com,16020,1514343549993, memstoreSize=538084641, blockingMemStoreSize=536870912
      	at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
      	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
      	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
       on ve0538.halxg.cloudera.com,16020,1514343549993, tracking started null, retrying after=10033ms, replay=524ops
      2017-12-26 19:24:37,183 INFO [ReadOnlyZKClient] org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: 0x015051a0 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.
      2017-12-26 19:24:39,122 WARN [htable-pool3-t6] org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: #2, table=IntegrationTestBigLinkedList, attempt=12/11 failed=524ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=IntegrationTestBigLinkedList,q\xC7\x1Cq\xC7\x1Cq\xC0,1514342757438.71ef1fbab1576588955f45796e95c08b., server=ve0538.halxg.cloudera.com,16020,1514343549993, memstoreSize=538084641, blockingMemStoreSize=536870912
      	at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4178)
      	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3799)
      	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3739)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:975)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:894)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2587)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      
      ...
      

      Fails task over and over. With server-killing monkeys.

      24Gs which should be more than enough.

      Had just finished a big compaction.

      Whats shutting us out? Why taking so long to flush? We seen stuck at limit so job fails.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stack Michael Stack
                Reporter:
                stack Michael Stack
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: