[HBASE-3303] Lower hbase.regionserver.handler.count from 25 back to 10

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.90.0
• Assignee: Jean-Daniel Cryans
• Reporter: Jean-Daniel Cryans
• Created: 02/Dec/10 22:01
• Updated: 20/Nov/15 12:43
• Resolved: 03/Dec/10 05:05

Description

With HBASE-2506 in mind, I tested a low-memory environment (2GB of heap) with a lot of concurrent writers using the default write buffer to verify if a lower number of handlers actually helps reducing the occurrence full GCs. Very unscientifically, at this moment I think it's safe to say that yes, it helps.

With the defaults, I saw a region server struggling more and more because the random inserters at some point started filling up all the handlers and were all BLOCKED trying to sync the WAL. It's safe to say that each of those clients carried a payload that the GC cannot get rid of and it's one that we don't account for (as opposed to MemStore and the block cache).

With a much lower setting of 5, I didn't see the situation.

It kind of confirms my hypothesis but I need to do more proper testing. In the mean time, in order to lower the onslaught of users that write to the ML complaining about either GCs or OOMEs, I think we should set the handlers back to what it was originally (10) for 0.90.0 and add some documentation about configuring hbase.regionserver.handler.count

I'd like to hear others' thoughts.