[CASSANDRA-19991] Unit test org.apache.cassandra.tools.TopPartitionsTest#testServiceTopPartitionsSingleTable is still flaky

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.15, 4.1.8, 5.0.2, 5.1
• Component/s: Test/unit
• Bug Category: Correctness - Test Failure
• Severity: Low
• Complexity: Low Hanging Fruit
• Discovered By: Unit Test
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra/commit/d939e40dfb59bebcf97b06b617e28031ff65fdc0

Test and Documentation Plan

an existing test is improved

• Assignee: Dmitry Konstantinov
• Reporter: Dmitry Konstantinov

Authors

Dmitry Konstantinov

Reviewers

Berenguer Blasi, Stefan Miklosovic

• Created: 09/Oct/24 21:33
• Updated: 01/Nov/24 16:22
• Resolved: 10/Oct/24 17:17

Description

org.apache.cassandra.tools.TopPartitionsTest#testServiceTopPartitionsSingleTable fails randomly and quite frequently on CI:

https://ci-cassandra.apache.org/view/Cassandra%204.1/job/Cassandra-4.1/521/testReport/org.apache.cassandra.tools/TopPartitionsTest/

https://ci-cassandra.apache.org/job/Cassandra-4.0/726/testReport/junit/org.apache.cassandra.tools/TopPartitionsTest/

Based on the source code - the most probably cause of this flakiness is async logic when we submit a sample here: org.apache.cassandra.metrics.Sampler#addSample - the submitted task can executed after we invoke finishLocalSampling(..) within the test.

The failure can be emulated by adding a synthetic delay to this logic like this:

public void addSample(final T item, final int value)
{
    if (isEnabled())
        samplerExecutor.submit(() -> {
            // ------ synthetic delay -----
            try
            {
                Thread.sleep(10000);
            } catch (Exception e) {};
            // ----------------------- -----
            insert(item, value);
        });
}

Issue Links

• duplicates: CASSANDRA-17798 Flaky org.apache.cassandra.tools TopPartitionsTest testServiceTopPartitionsSingleTable
• relates to: CASSANDRA-17455 Flaky org.apache.cassandra.tools TopPartitionsTest testServiceTopPartitionsSingleTable
• relates to: CASSANDRA-14360 Allow nodetool toppartitions without specifying table
• links to: GitHub Pull Request #3608
• links to: GitHub Pull Request #3609
• links to: GitHub Pull Request #3612