Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Unlike Counters and Gauges, Timers (SlidingTimeWindowReservoir) retain individual data points over a sliding window.
Most timers are instantiated on the order of the number of tasks. With the high level API it's per operator per task. So the number of data points used for each timer is:
NumOperatorsPerTask * NumTasks * TimerCollisionBuffer * TimerWindowSize * %WindowFill.
Values for a typical application could be:
4 * 128 * 1 * 300000 * 1 = 154M time-value pairs if the window is always filled (e.g. because we process a message every ms)
The timers are useful, but they're resource hungry.
SAMZA-834 reduced the memory usage by changing the default collision buffer from 256
Users can also disable the timers using the config:
metrics.timer.enabled=false
But the goal of this ticket is to explore ways to further reduce the memory overhead of timers so users don't have to turn them off.
A naive way is to reduce the window size, but ideally a more intelligent or coarser-grained (bucketed) sliding window implementation could be employed.