Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.0.0
-
None
-
None
Description
Once we have a storage engine and even if it is just a few metrics we want to add in an API to query the metrics.
It should be able to do simple aggregations (Min, Max, Mean, and Sum) across different time ranges, with different roll up windows. It should also be able to filter based off of things like topology id, component name, component id, stream name, and host name. It is fine to have some of these required, such as topology id, at the beginning.
It should be able to return results either all together or split up by the different topology ids, component names, component ids, stream names, or host names.
For Example the UI shows things like
SUM of
{tuples_emitted, tuples_transferred, acked, failed} in the last {10 mins, 3 hours, 1 day, all time} for all spouts a.k.a a set of component names that we got from the topology itself.AVERAGE of {complete_latency} in the last {10 mins, 3 hours, 1 day, all time} for a set of component names.
SUM of {tuples_emitted, tuples_transferred, acked, failed}
in the last
{10 mins, 3 hours, 1 day, all time} for all components separated by each component name.AVERAGE of {complete_latency} in the last {10 mins, 3 hours, 1 day, all time}
for all components separated by each component name.
And there are others too.
The API should be open to expansion for other types of filters, aggregates, and even data types. The following are not things that should be implemented, but are things that we should think about how we can leave the API open for expansion alter on so we could support them in a clean way.
Specifically some filters I would like to see in the future include topology name or even better a basic pattern we can use to match topology ids and the owner of a topology.
I would also like to eventually support percentile sketches as a data type so instead of just having an average of the 99th%ile collected by each component, we get an much more accurate approximation of what the 99th%ile really is in aggregate.