Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1967

Umbrella JIRA for node density improvements

    XMLWordPrintableJSON

Details

    Description

      For the Kudu 1.4 release, I'll be working to improve node density.

      Here's a brief primer on Kudu's scalability targets today:

      1. We recommend no more than 4 TB of total data per node. This is specific to Kudu data blocks, so this data is post-encoding and post-compression.
      2. We recommend no more than 1000 partitions (post-replication) per node.
      3. We recommend no more than 100 nodes per cluster.
      4. We recommend no more than 60 partitions per table per tserver.

      For 1.4, here's what we'd like to achieve:

      1. Up to 16 TB of total data per node. Maybe even 48 TB, if possible.
      2. Up to 100 "hot" partitions per node. In this context, "hot" means partitions that are actively servicing writes.
      3. Thousands of "cold" partitions per node. Put another way, it should be drastically cheaper to serve "cold" partitions than it is today.
      4. Maintain the "100 nodes per cluster" limit.
      5. Remove the "no more than 60 partitions per table per node" limit.

      I'll be linking various interesting JIRAs into this one, and I'll document, for each one, which aspect of data scalability it affects.

      Attachments

        Activity

          People

            Unassigned Unassigned
            adar Adar Dembo
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated: