Details
-
Improvement
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
Operability
-
Low Hanging Fruit
-
All
-
None
-
Description
During expansion by Dc, when we issue nodetool/rebuild from new dc to rebuild the data from other DCs. If src-dc is not passed explicitly, then C* tries to rebuild the data from the same (new dc) dc.
We don’t exclude other nodes in the same DC. Only down sources and the local node itself are excluded.
```
// We're always filtering out a local node and down sources
addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(failureDetector));
addSourceFilter(new RangeStreamer.ExcludeLocalNodeFilter());
```
We should fix nodetool/rebuild to exclude the local DC (from where we’re executing the command) while issuing nodetool/rebuild without passing src dc
Example:
in a 3 DC cluster,
ks1 has DC1, DC2
ks2 has DC1, DC2, DC3
ks3 has DC2
now, we add a new DC [DC4] and configured it to all 3 keyspaces.
if we run rebuild with src DC as DC1, ks3 will fail as it does not have DC1.
Now, without src DC, the expectation is rebuild would auto pick up DCs for each keyspace (let's say ks1: DC1, ks2: DC1, ks3: DC2) and would never fail due to under-replicated keyspaces.
The issue with this approach (without src dc) is that, DC4 is getting picked up during rebuild (as src), but DC4 does not have any data yet!
so, with the patch (ignore local dc flag), DC4 can be filtered out and let the database pick up the right dc for each keyspace [from existing 3 DCs].
– this is what is the expectation after the patch.
