[CASSANDRA-17658] Test Failures: org.apache.cassandra.metrics.KeyspaceMetricsTest.testMetricsCleanupOnDrop

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1-beta1, 5.0-alpha1, 5.0
• Component/s: Test/unit
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.1-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/c1d89c32d27921d1f77f05d29ee248b8922a4c76

Test and Documentation Plan

Repeated runs for the flaky test

• Assignee: Jacek Lewandowski
• Reporter: Andres de la Peña

Authors

Jacek Lewandowski

Reviewers

Andres de la Peña, Ekaterina Dimitrova

• Created: 24/May/22 18:25
• Updated: 12/Sep/23 13:03
• Resolved: 12/Aug/22 12:03

Description

The test org.apache.cassandra.metrics.KeyspaceMetricsTest.testMetricsCleanupOnDrop seems to be flaky on CircleCI, although I haven't seen it failing on Jenkins.

It was first detected during CASSANDRA-17509, on this run:

junit.framework.AssertionFailedError: org.apache.cassandra.metrics.keyspace.AdditionalWrites.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AllMemtablesLiveDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AllMemtablesOffHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AllMemtablesOnHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AntiCompactionTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.BloomFilterDiskSpaceUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.BloomFilterOffHeapMemoryUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.BytesValidated.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasCommitLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasCommitTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasPrepareLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasPrepareTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasProposeLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasProposeTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ClientTombstoneAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ClientTombstoneWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ColUpdateTimeDeltaHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CompressionMetadataOffHeapMemoryUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CoordinatorReadSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CoordinatorReadSizeAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CoordinatorReadSizeWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.IdealCLWriteLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.IdealCLWriteTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.IndexSummaryOffHeapMemoryUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LiveDiskSpaceUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LiveScannedHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LocalReadSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LocalReadSizeAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LocalReadSizeWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableColumnsCount.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableLiveDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableOffHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableOnHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableSwitchCount.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.PartitionsValidated.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.PendingCompactions.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.PendingFlushes.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RangeLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RangeTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ReadLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ReadTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairJobsCompleted.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairJobsStarted.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairPrepareTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairSyncTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataInconsistenciesConfirmed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataInconsistenciesUnconfirmed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataTrackingOverreadRows.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataTrackingOverreadTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RowIndexSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RowIndexSizeAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RowIndexSizeWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SSTablesPerReadHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SpeculativeFailedRetries.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SpeculativeInsufficientReplicas.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SpeculativeRetries.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.TombstoneScannedHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.TotalDiskSpaceUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ValidationTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ViewLockAcquireTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ViewReadTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.WriteFailedIdealCL.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.WriteLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.WriteTotalLatency.keyspacemetricstest_metrics_cleanup expected:<0> but was:<65>
	at org.apache.cassandra.metrics.KeyspaceMetricsTest.testMetricsCleanupOnDrop(KeyspaceMetricsTest.java:74)

The failure can be reproduced on the multiplexer with 1000 iterations:

.circleci/generate.sh -m \
  -e REPEATED_UTEST_TARGET=testsome \
  -e REPEATED_UTEST_COUNT=1000 \
  -e REPEATED_UTEST_CLASS=org.apache.cassandra.metrics.KeyspaceMetricsTest

Branch	CI
4.0	j8 j11
4.1	j8 j11
trunk	j8 j11

The failure can be reproduced on 4.1 and trunk, but apparently not in 4.0. Flakiness seems below 10%.

Issue Links

• Discovered while testing: CASSANDRA-17509 Add Guardrail to disable GROUP BY functionality
• is caused by: CASSANDRA-17071 Relax schema synchronization when opening a keyspace
• relates to: CASSANDRA-17819 Test failure: org.apache.cassandra.distributed.test.SchemaTest.schemaReset
• links to: PR 4.1
• links to: PR trunk