[AMBARI-15355] After EU from 2.2 -> 2.3 HBase region server started failed with memory error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.2
• Fix Version/s: 2.2.2
• Component/s: ambari-server
• Assignee: Dmitry Lysnichenko
• Reporter: Dmitry Lysnichenko
• Created: 09/Mar/16 16:29
• Updated: 11/Mar/16 12:29
• Resolved: 09/Mar/16 18:49

Description

On one of the clusters i did EU from 2.2.x to 2.3.x.

During upgrade there were problems with HBase service checks for region servers and thus upgrade is paused.

Region server start is failing with error

2016-03-03 19:55:31,203 ERROR [regionserver:16020] regionserver.HRegionServer: Failed init
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65)
at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:217)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231)
at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899)
at java.lang.Thread.run(Thread.java:745)
2016-03-03 19:55:31,206 FATAL [regionserver:16020] regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort itself immediately
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65)
at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:217)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637)
at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231)
at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899)
at java.lang.Thread.run(Thread.java:745)
2016-03-03 19:55:35,138 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT

This was seen on the following cluster: https://s.c:8443/#/main/services/HBASE/configs

Issue Links

• links to: review_request_