[HBASE-11165] Scaling so cluster can host 1M regions and beyond (50M regions?)

• Type: Brainstorming
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 14/May/14 03:23
• Updated: 17/May/20 13:03

Description

This discussion issue comes out of "Co-locate Meta And Master HBASE-10569" and comments on the doc posted there.

A user – our Francis Liu – needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end).

More detail to follow.

Issue Links

• is a parent of: HBASE-11288 Splittable Meta
• relates to: HBASE-11267 Dynamic metrics2 metrics may consume large amount of heap memory
• links to: Scalable Meta Notes

Sub-Tasks

• BulkDisabler should use a bulk RPC call for opening regions (just like BulkAssigner): Open
• Flush table cmd should make RPC calls to region server in parallel: Open
• Speedup balance: Open
• Unlock RegionStates: Open
• Parallelize meta scan during startup: Open
• Enhance remote meta updates: Closed
• ZK-less region assignment: Closed
• Backport ZK-less region assignment to 0.98: Closed
• ClusterStatus (heartbeat) is too bulky: Open
• RegionStates.getRegionAssignments() gets stuck on clone: Closed
• Meta region location should be cached: Closed
• TableZnode should be cached: Closed
• Bring back root table: Open
• Hierarchical Layout for Humongous Tables: Open
• Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS: Open
• Make RegionSizeCalculator scalable: Closed