[HBASE-12369] Warn if hbase.bucketcache.size too close or equal to MaxDirectMemorySize

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Incomplete
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: Esteban Gutierrez
• Created: 28/Oct/14 23:55
• Updated: 17/Jun/22 17:43
• Resolved: 17/Jun/22 17:43

Description

Our ref guide currently says that its required to leave some room from the DirectMemory. However if hbase.bucketcache.size is too close or equal to MaxDirectMemorySize it can trigger OOMEs:

2014-10-28 16:14:41,585 INFO  [master//172.16.0.101:16020] util.ByteBufferArray: Allocating buffers total=5.00 GB, sizePerBuffer=4 MB, count=1280, direct=true
2014-10-28 16:14:41,604 INFO  [172.16.0.101:16020.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 1, slept for 99 ms, expecting minimum of 2, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2014-10-28 16:14:43,144 INFO  [172.16.0.101:16020.activeMasterManager] master.ServerManager: Waiting for region servers count to settle; currently checked in 1, slept for 1639 ms, expecting minimum of 2, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2014-10-28 16:14:44,057 INFO  [master//172.16.0.101:16020] regionserver.HRegionServer: STOPPED: Failed initialization
2014-10-28 16:14:44,058 ERROR [master//172.16.0.101:16020] regionserver.HRegionServer: Failed init
java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
	at org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65)
	at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47)
	at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:310)
	at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:218)
	at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:513)
	at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:536)
	at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:213)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1259)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:818)
	at java.lang.Thread.run(Thread.java:724)

It would be helpful to print a warn message that hbase.bucketcache.size too close or equal to MaxDirectMemorySize.