[CASSANDRA-4119] Support multiple non-consecutive tokens per host (virtual nodes)

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Labels: virtualnodes vnodes
• Assignee: Sam Overton
• Reporter: Sam Overton

Authors

Sam Overton

• Created: 05/Apr/12 13:37
• Updated: 16/Apr/19 09:32
• Resolved: 15/Feb/13 05:59

Description

This is the parent ticket for the virtual nodes implementation which was proposed here: http://www.mail-archive.com/[e-mail]/msg03837.html and discussed in the subsequent thread.

The goals of this ticket are:

• reduced operations complexity for scaling up/down
• reduced rebuild time in event of failure
• evenly distributed load impact in the event of failure
• evenly distributed impact of streaming operations
• more viable support for heterogeneity of hardware

The intention is that this can be done in a way which is

• fully backwards-compatible
• optionally enabled

The latter of these can be trivially achieved by setting the number of tokens per host to 1, to reproduce the existing behaviour.

Implementation detail can be added and discussed in the sub-tickets, but here is an overview of the proposed changes:

• TokenMetadata will allow multiple tokens per host
• Hosts will be referred to by a UUID instead of token (e.g. in Gossip, when storing hints, etc.)
• A bootstrapping node can get multiple tokens from initial_token (comma separated) or by random allocation
• NetworkTopologyStrategy will be extended to be aware of virtual nodes so that replicas are not placed on the same host (similar to racks now)
• Repairs will be staggered similar to CASSANDRA-3721
• Nodetool operations will be virtual-node aware, while maintaining backwards compatibility (ie. existing scripts won't have to change)
• Upgrade will be a standard rolling upgrade, with optional rolling migration to full vnode support

Issue Links

• is blocked by: CASSANDRA-3881 reduce computational complexity of processing topology changes

Sub-Tasks

• Gossip identifies hosts by UUID: Resolved
• TokenMetadata supports multiple tokens per host: Resolved
• Bootstrap and decommission with multiple ranges for vnodes: Resolved
• staggered repair for multiple ranges: Resolved
• Update nodetool for vnodes: Resolved
• write tests for vnodes: Resolved
• migration support for vnodes: Resolved
• Binary encoding of vnode tokens: Resolved
• shuffle utility for vnodes: Resolved
• implement token relocation: Resolved