[HBASE-10643] Failure in RS when using large size bucketcache

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.98.0, 0.96.0
• Component/s: BlockCache, regionserver
• Labels: beginner
• Assignee: Unassigned
• Reporter: Biju Nair
• Created: 28/Feb/14 20:14
• Updated: 17/Jun/22 05:09
• Resolved: 26/Jul/17 21:40

Description

When RS is brought up with XX:MaxDirectMemorySize of 22GB or higher, RS fails after a successful start. From the RS logs it looks like the bucketCache memory allocation is taking more time makes the RS considered dead by ZK. One option to fix the problem would be to allocate the bucketCache before registering with ZK.

2014-02-28 18:54:42,967 WARN [regionserver60020.compactionChecker] util.Sleeper: We slept 33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2014-02-28 18:54:42,967 WARN [regionserver60020.periodicFlusher] util.Sleeper: We slept 33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2014-02-28 18:54:42,967 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 23988ms
GC pool 'ParNew' had collection(s): count=1 time=24432ms
2014-02-28 18:54:43,006 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING region server bbg-master2.bbg-test.hdp,60020,1393628951236: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing bbg-master2.bbg-test.hdp,60020,1393628951236 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:341)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)

Issue Links

• is superceded by: HBASE-17738 BucketCache startup is slow