[CASSANDRA-17955] Race condition on repair snapshots

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.8, 4.1-rc1, 4.1, 5.0
• Component/s: Consistency/Repair, Local/Snapshots
• Labels: 4.0
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra/commit/35ef5b99577ef8b04b8d4b326154775f510ade42

Test and Documentation Plan

ci

• Assignee: Stefan Miklosovic
• Reporter: Cameron Zemek

Authors

Stefan Miklosovic

• Created: 10/Oct/22 00:39
• Updated: 22/Nov/22 15:42
• Resolved: 27/Oct/22 14:52

Description

If an endpoint is convicted and that endpoint is a coordinator then ActiveRepairService::removeParentRepairSession is called.

The issue is that this occurs on clearSnapshotExecutor and can happen while RepairMessageVerbHandler is in process of taking a snapshot. So then you get a race condition and clearSnapshot will throw a java.nio.file.DirectoryNotEmptyException

 

public static void deleteRecursiveWithThrottle(File dir, RateLimiter rateLimiter)
{
    if (dir.isDirectory())
    {
        String[] children = dir.list();
        for (String child : children)
            deleteRecursiveWithThrottle(new File(dir, child), rateLimiter);
    }

    // The directory is now empty so now it can be smoked
    deleteWithConfirmWithThrottle(dir, rateLimiter);
} 

Due to the directory not being empty when it goes to remove the directory at the end.