[CASSANDRA-11752] histograms/metrics in 2.2 do not appear recency biased

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.8, 3.0.9, 3.8
• Component/s: Legacy/Core
• Labels: metrics
• Severity: Normal
• Assignee: Per Otterström
• Reporter: Chris Burroughs

Authors

Per Otterström

Reviewers

T Jake Luciani

• Created: 11/May/16 17:44
• Updated: 16/Apr/19 09:30
• Resolved: 10/Aug/16 18:45

Description

In addition to upgrading to metrics3, CASSANDRA-5657 switched to using a custom histogram implementation. After upgrading to Cassandra 2.2 histograms/timer metrics are not suspiciously flat. To be useful for graphing and alerting metrics need to be biased towards recent events.

I have attached images that I think illustrate this.

• The first two are a comparison between latency observed by a C* 2.2 (us) cluster shoring very flat lines and a client (using metrics 2.2.0, ms) showing server performance problems. We can't rule out with total certainty that something else isn't the cause (that's why we measure from both the client & server) but they very rarely disagree.
• The 3rd image compares jconsole viewing of metrics on a 2.2 and 2.1 cluster over several minutes. Not a single digit changed on the 2.2 cluster.