[OAK-2596] more (jmx) instrumentation for observation queue

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.0.12
• Fix Version/s: 1.1.8
• Component/s: core
• Labels: monitoring observation
• Assignee: Michael Dürig
• Reporter: Stefan Egli
• Created: 09/Mar/15 15:51
• Updated: 12/Apr/15 14:52
• Resolved: 26/Mar/15 16:12

Description

While debugging issues with the observation queue it would be handy to have more detailed information available. At the moment you can only see one value wrt length of the queue: that is the maximum of all queues. It is unclear if the queue is that long for only one or many listeners. And it is unclear from that if the listener is slow or the engine that produces the events for the listener.

So I'd suggest to add the following details - possible exposed via JMX? :

1. add queue length details to each of the observation listeners
2. have a history of the last, eg 1000 events per listener showing a) how long the event took to be created/generated and b) how long the listener took to process. Sometimes averages are not detailed enough so such a in-depth information might become useful. (Not sure about the feasibility of '1000' here - maybe that could be configurable though - just putting the idea out here).
3. have some information about whether a listener is currently 'reading events from the cache' or whether it has to go to eg mongo
4. maybe have a 'top 10' listeners that have the largest queue at the moment to easily allow navigation instead of having to go through all (eg 200) listeners manually each time.

Issue Links

• is blocked by: JCR-3859 Extend EventListenerMBean to report time series for observation processing
• is blocked by: OAK-2632 Upgrade Jackrabbit dependency to 2.10.0
• is related to: OAK-2755 Consolidated JMX view of all EventListener related statistics