Details
-
New Feature
-
Status: Open
-
Normal
-
Resolution: Unresolved
-
None
-
Operability
-
Normal
-
All
-
None
Description
There is a disconnect with latency clients experience and the latency reported by Cassandra. For example read latency only measures the latency of the StorageProxy::readRows call.
None of the time spent sitting in the Native Transport queue is measured. Neither is any of the time for writing the response back to the channel.
Dispatcher processRequest keeps track of when it first starts processing the request but best I can tell this is only used in tracking for timeouts.
It would be useful for tracking down cause of high client latency if there were more detailed Cassandra metrics around it.
I have attached a patch that adds latency tracking higher in the call stack. Starting timer from before it is put into the Native Transport Request executor. The patch gives 3 different metrics per Request type:
delay - measures time from when it is submitted to NTR pool until it call processRequest
process - time spent in the Dispatcher processRequest call
total - time from when first submitted to NTR pool until the response has been flushed
This patch may not be cleanest or best way of doing this but hopefully gives an idea of what I think would be useful addition that will help operators diagonse latency issues.