[CASSANDRA-17808] Optionally avoid hint transfer during decommission

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-alpha1, 5.0
• Component/s: Consistency/Hints
• Change Category: Operability
• Complexity: Normal
• Platform: All

Source Control Link

https://github.com/apache/cassandra/commit/d6aee7e08c658db9d394a6b7e3e27791b4d6854f

Test and Documentation Plan

A new test verifying the absence of hint transfer during decommission has been added.

• Assignee: Caleb Rackliffe
• Reporter: Caleb Rackliffe

Authors

Caleb Rackliffe

Reviewers

Jon Meredith, Stefan Miklosovic

• Created: 10/Aug/22 16:20
• Updated: 04/Apr/24 15:08
• Resolved: 08/Sep/22 21:39

Description

Both because they aren’t strictly necessary to maintain consistency, and because throttling induced by their rate-limiter (see hinted_handoff_throttle) may stall progress, transferring hints during decommission (specifically unbootstrap) rather than just pausing, disabling, and truncating them probably doesn’t make sense. The only other concern would be the BatchLog, which nominally depends on hint delivery to maintain its "guarantees". However, during BatchLog replay on unbootstrap, ReplayingBatch ignores batches older the gcgs anyway.

Here's a proposal from aleksey that might strike a reasonable balance:

1.) We continue to transfer hints by default during decommission, but at a higher rate. We could, for instance, stop having DispatchHintsTask divide its effective rate by the number of nodes in the cluster.

int nodesCount = Math.max(1, StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() * 1024.0 / nodesCount;
this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE : throttleInBytes);

2.) We provide an option to simply avoid transferring hints during unbootstrap. Even this would only take the BatchLog from "best effort" to "slightly less effort"

Issue Links

• is related to: CASSANDRA-16679 HintedHandoffAddRemoveNodesTest is failing
• is related to: CASSANDRA-5128 Stream hints on decommission