[KUDU-1967] Umbrella JIRA for node density improvements

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.3.0
• Component/s: fs, master, tablet, tserver
• Labels: data-scalability roadmap-candidate
• Assignee: Unassigned
• Reporter: Adar Dembo
• Created: 11/Apr/17 00:15
• Updated: 31/Mar/20 07:24

Description

For the Kudu 1.4 release, I'll be working to improve node density.

Here's a brief primer on Kudu's scalability targets today:

1. We recommend no more than 4 TB of total data per node. This is specific to Kudu data blocks, so this data is post-encoding and post-compression.
2. We recommend no more than 1000 partitions (post-replication) per node.
3. We recommend no more than 100 nodes per cluster.
4. We recommend no more than 60 partitions per table per tserver.

For 1.4, here's what we'd like to achieve:

1. Up to 16 TB of total data per node. Maybe even 48 TB, if possible.
2. Up to 100 "hot" partitions per node. In this context, "hot" means partitions that are actively servicing writes.
3. Thousands of "cold" partitions per node. Put another way, it should be drastically cheaper to serve "cold" partitions than it is today.
4. Maintain the "100 nodes per cluster" limit.
5. Remove the "no more than 60 partitions per table per node" limit.

I'll be linking various interesting JIRAs into this one, and I'll document, for each one, which aspect of data scalability it affects.

Sub-Tasks

• bootstrap should not replay logs that are known to be fully flushed: Open
• Explore reducing number of data blocks by tuning existing parameters: Open
• Explore ways to reduce maintenance manager CPU load: Open
• Coalesce RPCs destined for the same server: Open
• Skip entire WAL files or sections of WAL files that do not need to be replayed: Open