[KUDU-3115] Improve scalability of Kudu masters

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: master
• Labels: scalability
• Assignee: Unassigned
• Reporter: Alexey Serbin
• Created: 04/May/20 19:08
• Updated: 20/Apr/21 23:29

Description

Currently, multiple masters in a multi-master Kudu cluster are used only for high availability & fault tolerance use cases, but not for sharing the load among the available master nodes. For example, Kudu clients detect current leader master upon connecting to the cluster and send all their subsequent requests to the leader master, so serving many more clients require running masters on more powerful nodes. Current design assumes that masters store and process the requests for metadata only, but that makes sense only up to some limit on the rate of incoming client requests.

It would be great to achieve better 'horizontal' scalability for Kudu masters.

Issue Links

• relates to: KUDU-500 Changes to SysTables should be queryable on follower leaders upon replication
• relates to: KUDU-2955 kudu-master: separate RPC service queues for TSHeartbeat from client-facing RPCs
• relates to: KUDU-3135 Add Client Metadata Tokens
• relates to: KUDU-1802 Deserializing scan tokens should avoid round-trip to master
• relates to: KUDU-3242 Investigate performance of GetTableSchema when authz tokens enabled