Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-10643

Failure in RS when using large size bucketcache

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.98.0, 0.96.0
    • Fix Version/s: None
    • Component/s: BlockCache, regionserver
    • Labels:

      Description

      When RS is brought up with XX:MaxDirectMemorySize of 22GB or higher, RS fails after a successful start. From the RS logs it looks like the bucketCache memory allocation is taking more time makes the RS considered dead by ZK. One option to fix the problem would be to allocate the bucketCache before registering with ZK.

      2014-02-28 18:54:42,967 WARN [regionserver60020.compactionChecker] util.Sleeper: We slept 33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
      2014-02-28 18:54:42,967 WARN [regionserver60020.periodicFlusher] util.Sleeper: We slept 33496ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
      2014-02-28 18:54:42,967 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 23988ms
      GC pool 'ParNew' had collection(s): count=1 time=24432ms
      2014-02-28 18:54:43,006 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING region server bbg-master2.bbg-test.hdp,60020,1393628951236: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing bbg-master2.bbg-test.hdp,60020,1393628951236 as dead server
      at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:341)
      at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                gsbiju Biju Nair
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: