[SLING-5965] Metrics and a Health-Check for Scheduler to detect long-running Quartz-Jobs

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Commons Scheduler 2.5.0
• Fix Version/s: Commons Scheduler 2.7.0
• Component/s: Commons
• Assignee: Stefan Egli
• Reporter: Stefan Egli
• Created: 16/Aug/16 10:03
• Updated: 26/Sep/19 12:40
• Resolved: 22/Aug/17 09:12

Description

Sling Scheduler jobs (aka Quartz-Jobs) should typically be fast running jobs. They are served from a thread-pool and should occupy that thread only for a short amount of time.

If there are 'misbehaving' quartz-jobs that run for a very long time, they start to occupy threads from that thread-pool, thus have an influence on the performance of other scheduled/quartz-jobs.

We should have metrics (using sling.commons.metrics) that provide information about internas of Sling Scheduler, such as average, max etc duration of scheduled jobs, as well as how many jobs are currently running and since when was the oldest job running.

Based on this, a Health-Check can monitor the 'oldest job running' metric and flag critical when eg the oldest job is older than 60'000ms (configurable, default).

Issue Links

• is blocked by: SLING-5424 MBeanServer reference in MetricServiceImpl should be made optional
• is related to: SLING-5966 Add Gauge support to metrics