[HBASE-5970] Improve the AssignmentManager#updateTimer and speed up handling opened event

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.95.0
• Component/s: master
• Hadoop Flags: Reviewed
• Assignee: Chunhui Shen
• Reporter: Chunhui Shen
• Created: 09/May/12 04:44
• Updated: 23/Sep/13 18:30
• Resolved: 31/May/12 05:48

Description

We found handing opened event very slow in the environment with lots of regions.

The problem is the slow AssignmentManager#updateTimer.

We do the test for bulk assigning 10w (i.e. 100k) regions, the whole process of bulk assigning took 1 hours.

2012-05-06 20:31:49,201 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 100000 region(s) round-robin across 5 server(s)
2012-05-06 21:26:32,103 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done

I think we could do the improvement for the AssignmentManager#updateTimer: Make a thread do this work.
After the improvement, it took only 4.5mins

2012-05-07 11:03:36,581 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 100000 region(s) across 5 server(s), retainAssignment=true
2012-05-07 11:07:57,073 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done

Issue Links

• is duplicated by: HBASE-5337 AM.updateTimers() delays the timeout monitor from assigning regions.
• is related to: HBASE-5843 Improve HBase MTTR - Mean Time To Recover
• relates to: HBASE-7038 Port HBASE-5970 Improve the AssignmentManager#updateTimer and speed up handling opened event to 0.94