Some tests results:
I tested the following scenarios, on a local machine, a pseudo
distributed cluster with ZooKeeper and HBase writing in a ram drive,
no datanode nor namenode, with 2 region servers, and one empty table
with 10000 regions, 5K on each RS. Versions taken monday 2nd
1) Clean stop of one RS; wait for all regions to become online again:
0.92: ~800 seconds
0.96: ~13 seconds
=> Huge improvement, hopefully from stuff like
HBASE-5970 and HBASE-6109.
1.1) As above with 2Mb memory per server
Results as 1)
=> Results don't depend on any GC stuff (memory reported is around 200 Mb)
2) Kill -9 of a RS; wait for all regions to become online again:
=> The 180s gap comes from
HBASE-5844. For master, HBASE-5926 is not tested but should bring similar results.
3) Start of the cluster after a clean stop; wait for all regions to
0.94: ~1023s (tested once only)
=> The benefit is visible at startup
=> This does not come from something implemented for 0.94
4) As 3) But with HBase on a local HD
0.92: ~1044s (tested once only)
0.96: ~28s (tested once only)
=> Similar results. Seems that HBase i/o was not and is not becoming the bottleneck.
5) As 1) With 4RS instead of 2
=> Twice faster in both cases. Scales with the number of RS with both versions on this minimalistic test.
6) As 3) But with ZK on a local HD
Impossible to get something consistent here. Machine and test dependent.
The most credible result was similar to 2).
From ZK mailing list or ZOOKEEPER-866 is seems that what we should expect.
7) With 2 RS, Insert 20M simple puts; then kill -9 the second one. See how long it takes to have all the regions available.
0.92) 180s detection time+ then hangs twice out of 2 tests.
0.96) 14s (hangs once out of 3)
=> There's a bug
=> Test to be changed to get a real difference when we need to replay the wal.