[CASSANDRA-13840] GraphiteReporter stops reporting from Memory concurrent access

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Won't Fix
• Fix Version/s: 2.1.x
• Component/s: Legacy/Observability
• Severity: Normal
• Assignee: Michael Semb Wever
• Reporter: Michael Semb Wever

Authors

Michael Semb Wever

• Created: 04/Sep/17 05:16
• Updated: 21/Aug/21 16:21
• Resolved: 21/Aug/21 16:21

Description

Gauges in the GraphiteReporter can, when getting compression offheap size, trip over the assertion in Memory.size(). As the GraphiteReporter only handles Exceptions this crashes the reporter and it sends no more metrics.

benedict described it here and jbellis confirmed that it's now ok to remove the assertion.

java.lang.AssertionError: null
        at org.apache.cassandra.io.util.Memory.size(Memory.java:359) ~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.io.compress.CompressionMetadata.offHeapSize(CompressionMetadata.java:167) ~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.io.sstable.SSTableReader.getCompressionMetadataOffHeapSize(SSTableReader.java:1315) ~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.metrics.ColumnFamilyMetrics$30.value(ColumnFamilyMetrics.java:573) ~[apache-cassandra-2.1.12.jar:2.1.12]
        at org.apache.cassandra.metrics.ColumnFamilyMetrics$30.value(ColumnFamilyMetrics.java:568) ~[apache-cassandra-2.1.12.jar:2.1.12]
        at com.yammer.metrics.reporting.GraphiteReporter.processGauge(GraphiteReporter.java:309) ~[metrics-graphite-2.2.0.jar:na]
        at com.yammer.metrics.reporting.GraphiteReporter.processGauge(GraphiteReporter.java:26) ~[metrics-graphite-2.2.0.jar:na]
        at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28) ~[metrics-core-2.2.0.jar:na]
        at com.yammer.metrics.reporting.GraphiteReporter.printRegularMetrics(GraphiteReporter.java:251) ~[metrics-graphite-2.2.0.jar:na]
        at com.yammer.metrics.reporting.GraphiteReporter.run(GraphiteReporter.java:216) ~[metrics-graphite-2.2.0.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]