[SLING-4460] QuartzScheduler component is registered resulting in sling jobs not being sent to job consumer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Commons Scheduler 2.4.4
• Fix Version/s: Commons Scheduler 2.4.6
• Component/s: Commons
• Environment: 3.3.4-11.omtr.el6.x86_64 GNU/Linux, java version "1.7.0_60"
• Assignee: Carsten Ziegeler
• Reporter: Ken Russell
• Created: 27/Feb/15 05:36
• Updated: 04/Mar/15 14:24
• Resolved: 27/Feb/15 17:58

Description

Sling jobs were not being processed on an instance due to the QuartzScheduler component being in a registered state (see attached image for the component’s state). The Sling job processor was not sending the job to our job consumer. From our error logs we could see that the job was created but the job consumer never received it. Looking at the crx we could see the job under /var/eventing. Our bundles & components were all active (we explicitly checked the component that consumes the job).

Occasionally we see this problem on a restart of a production instance. This is an instance on our QA environment (we are hoping they are the same but will continue to monitor on subsequent production deployments).

We noticed in the log file lots of exceptions related to QuartzScheduler indicating that the “Scheduler is not available anymore.” Truncated exception:

26.02.2015 22:05:16.904 *WARN* [<redacted> [1424988304541] POST /system/console/bundles/342 HTTP/1.1] org.apache.sling.discovery.impl.DiscoveryServiceImpl sendTopologyEvent: handler threw exception. handler: org.apache.sling.event.impl.jobs.config.JobManagerConfiguration@651e2ff3, exception: java.lang.IllegalStateException: Scheduler is not available anymore. java.lang.IllegalStateException: Scheduler is not available anymore.
        at org.apache.sling.commons.scheduler.impl.QuartzScheduler.scheduleJob(QuartzScheduler.java:634)
        at org.apache.sling.commons.scheduler.impl.QuartzScheduler.schedule(QuartzScheduler.java:578)
        at org.apache.sling.commons.scheduler.impl.SchedulerServiceFactory.schedule(SchedulerServiceFactory.java:56)
        at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.startProcessing(JobManagerConfiguration.java:500)
        at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.handleTopologyEvent(JobManagerConfiguration.java:551)
        at org.apache.sling.discovery.impl.DiscoveryServiceImpl.sendTopologyEvent(DiscoveryServiceImpl.java:192)
        at org.apache.sling.discovery.impl.DiscoveryServiceImpl.handlePotentialTopologyChange(DiscoveryServiceImpl.java:484)
        at org.apache.sling.discovery.impl.DiscoveryServiceImpl.bindPropertyProviderInteral(DiscoveryServiceImpl.java:272)

From grepping the error log we saw that the QuartzScheduler has been shutdown previously:

> cat error.log.2015-02-25 error.log | grep QuartzScheduler
25.02.2015 19:16:31.246 *INFO* [OsgiInstallerImpl] org.quartz.core.QuartzScheduler Scheduler ApacheSling_$_Tue_Feb_24_10:52:11_UTC_2015 shutting down.
25.02.2015 19:16:31.246 *INFO* [OsgiInstallerImpl] org.quartz.core.QuartzScheduler Scheduler ApacheSling_$_Tue_Feb_24_10:52:11_UTC_2015 paused.
25.02.2015 19:16:31.251 *INFO* [OsgiInstallerImpl] org.quartz.core.QuartzScheduler Scheduler ApacheSling_$_Tue_Feb_24_10:52:11_UTC_2015 shutdown complete.
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.scheduleJob(QuartzScheduler.java:634)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.schedule(QuartzScheduler.java:578)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.scheduleJob(QuartzScheduler.java:634)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.schedule(QuartzScheduler.java:578)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.scheduleJob(QuartzScheduler.java:634)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.schedule(QuartzScheduler.java:578)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.scheduleJob(QuartzScheduler.java:634)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.schedule(QuartzScheduler.java:578)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.scheduleJob(QuartzScheduler.java:634)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.schedule(QuartzScheduler.java:578)
	at org.apache.sling.commons.scheduler.impl.QuartzScheduler.scheduleJob(QuartzScheduler.java:634)

Stop-Starting the QuartzScheduler resulted in all the jobs being processed.