[CASSANDRA-9973] java.lang.IllegalStateException: Unable to compute when histogram overflowed

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.1
• Component/s: Legacy/Observability
• Severity: Normal
• Assignee: T Jake Luciani
• Reporter: Mark Manley

Authors

T Jake Luciani

Reviewers

Carl Yeksigian

• Created: 04/Aug/15 03:30
• Updated: 16/Apr/19 09:31
• Resolved: 14/Aug/15 14:27

Description

I recently, and probably mistakenly, upgraded one of my production C* clusters to 2.2.0. I am seeing these errors in the logs, followed by an intense period of garbage collection until the node, then the ring, becomes crippled:

ERROR [OptionalTasks:1] 2015-08-04 03:24:56,057 CassandraDaemon.java:182 - Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.IllegalStateException: Unable to compute when histogram overflowed
        at org.apache.cassandra.utils.EstimatedHistogram.percentile(EstimatedHistogram.java:179) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getValue(EstimatedHistogramReservoir.java:84) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.db.ColumnFamilyStore$3.run(ColumnFamilyStore.java:405) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

I am not sure if the GC instability is this or something else, but I though this histogram overflow issue was fixed in 2.1.3? Anyway, reporting now as a possible regression. Please let me know what I can provide in terms of information to help with this. Thanks!