[HBASE-2175] Investigate .META. slowdowns when more than 1 store files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Later
• Fix Version/s: 0.90.0
• Assignee: Unassigned
• Reporter: Jean-Daniel Cryans
• Created: 29/Jan/10 19:44
• Updated: 20/Nov/15 13:01
• Resolved: 05/Oct/10 22:44

Description

I'm currently testing Hadoop 0.21 with HBase trunk + HBASE-2066 by importing our main data set. After some time, probably because of log rolls which force flushes and a cluster restart, the .META. region begins to accumulate store files. I'm refreshing the master web UI a lot to see our insert speed and saw that 1) it was getting slower to refresh and 2) the import speed went down at the same time.

Having already seen something like that previously with 0.20, I forced a major compaction on .META. and immediately the refresh speed got 10 times better and the import throughput went 2x (tasks went from 20 min to 10 min).

Why is scanning and doing random reads from the client that slow when .META. has more than 1 store file? If it's a more fondamental speed issue, could we at least force major compactions on .META. when it grows so that the rest of the cluster doesn't get super slow? By the way, that operation takes less than 1 second since that region is so small.

Issue Links

• relates to: HBASE-2189 HCM trashes meta cache even when not needed