Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.2.2
-
None
Description
On one of the clusters i did EU from 2.2.x to 2.3.x.
During upgrade there were problems with HBase service checks for region servers and thus upgrade is paused.
Region server start is failing with error
2016-03-03 19:55:31,203 ERROR [regionserver:16020] regionserver.HRegionServer: Failed init java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) at org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65) at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:217) at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) at java.lang.Thread.run(Thread.java:745) 2016-03-03 19:55:31,206 FATAL [regionserver:16020] regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort itself immediately java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) at org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65) at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:217) at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) at java.lang.Thread.run(Thread.java:745) 2016-03-03 19:55:35,138 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT
This was seen on the following cluster: https://s.c:8443/#/main/services/HBASE/configs
Attachments
Attachments
Issue Links
- links to