Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1967

Umbrella JIRA for node density improvements

    XMLWordPrintableJSON

    Details

      Description

      For the Kudu 1.4 release, I'll be working to improve node density.

      Here's a brief primer on Kudu's scalability targets today:

      1. We recommend no more than 4 TB of total data per node. This is specific to Kudu data blocks, so this data is post-encoding and post-compression.
      2. We recommend no more than 1000 partitions (post-replication) per node.
      3. We recommend no more than 100 nodes per cluster.
      4. We recommend no more than 60 partitions per table per tserver.

      For 1.4, here's what we'd like to achieve:

      1. Up to 16 TB of total data per node. Maybe even 48 TB, if possible.
      2. Up to 100 "hot" partitions per node. In this context, "hot" means partitions that are actively servicing writes.
      3. Thousands of "cold" partitions per node. Put another way, it should be drastically cheaper to serve "cold" partitions than it is today.
      4. Maintain the "100 nodes per cluster" limit.
      5. Remove the "no more than 60 partitions per table per node" limit.

      I'll be linking various interesting JIRAs into this one, and I'll document, for each one, which aspect of data scalability it affects.

        Attachments

          Activity

            People

            • Assignee:
              adar Adar Dembo
              Reporter:
              adar Adar Dembo
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: