[HBASE-2506] Too easy to OOME a RS

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Labels: moved_from_0_20_5
• Assignee: Unassigned
• Reporter: Jean-Daniel Cryans
• Created: 30/Apr/10 20:45
• Updated: 11/Jun/22 23:33
• Resolved: 09/Feb/15 06:08

Description

Testing a cluster with 1GB heap, I found that we are letting the region servers kill themselves too easily when scanning using pre-fetching. To reproduce, get 10-20M rows using PE and run a count in the shell using CACHE => 30000 or any other very high number. For good measure, here's the stack trace:

2010-04-30 13:20:23,241 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError, aborting.
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2786)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.hbase.client.Result.writeArray(Result.java:478)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:312)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.write(HbaseObjectWritable.java:229)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:941)
2010-04-30 13:20:23,241 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=29, stores=29, storefiles=44, storefileIndexSize=6, memstoreSize=255,
 compactionQueueSize=0, usedHeap=926, maxHeap=987, blockCacheSize=1700064, blockCacheFree=205393696, blockCacheCount=0, blockCacheHitRatio=0

I guess the same could happen with largish write buffers. We need something better than OOME.

Issue Links

• relates to: HBASE-6728 [89-fb] prevent OOM possibility due to per connection responseQueue being unbounded
• relates to: HBASE-7023 Forward-port HBASE-6727 size-based HBaseServer callQueue throttle from 0.89fb branch