[CAMEL-7809] Quartz PollConsumerScheduler in a cluster tries to create duplicate triggers, fails

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.13.1
• Fix Version/s: 2.18.4, 2.19.0
• Component/s: camel-quartz2

Environment

identical CamelContexts deployed to each node in a cluster
Quartz 2.2.1 with JDBC JobStore and synchronization via DB

• Patch Info: Patch Available
• Estimated Complexity: Unknown
• Assignee: Claus Ibsen
• Reporter: Steffen Ryll
• Created: 12/Sep/14 17:12
• Updated: 28/Mar/17 09:48
• Resolved: 27/Mar/17 13:23

Description

I tested the patch developed in CAMEL-7663 and ran into another issue that prevents the CamelContext from starting up. The setup is the same as described in CAMEL-7663.
Upon startup I get following exception from Quartz on at least one cluster node:

13:23:36|INFO |component.quartz2.QuartzComponent|[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'|Shutting down scheduler. (will wait for all jobs to complete first.)
13:23:36|INFO |quartz.core.QuartzScheduler|[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'|Scheduler IPP_Integration_Scheduler-emerald-integration-web_$_fra...1410348212901 shutting down.
13:23:36|INFO |quartz.core.QuartzScheduler|[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'|Scheduler IPP_Integration_Scheduler-emerald-integration-web_$_fra...1410348212901 paused.
13:23:36|INFO |quartz.core.QuartzScheduler|[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'|Scheduler unregistered from name 'quartz:type=QuartzScheduler,name=IPP_Integration_Scheduler-emerald-integration-web,instance=fra...1410348212901' in the local MBeanServer.
2014-09-10-13:23:36|INFO |quartz.core.QuartzScheduler|[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'|Scheduler IPP_Integration_Scheduler-emerald-integration-web_$_fra...1410348212901 shutdown complete.
13:23:36|ERROR|component.servletlistener.CamelServletContextListener|[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'|Error starting CamelContext.
org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'scheduler.fax.in' and group: 'QuartzScheduledPollConsumerScheduler', because one already exists with this identification.
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1179)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$2.executeVoid(JobStoreSupport.java:1063)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3703)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3701)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3787)
	at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:93)
	at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1058)
	at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:886)
	at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:249)
	at org.apache.camel.pollconsumer.quartz2.QuartzScheduledPollConsumerScheduler.doStart(QuartzScheduledPollConsumerScheduler.java:187)
	at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
	at org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:74)
	at org.apache.camel.impl.ScheduledPollConsumer.doStart(ScheduledPollConsumer.java:499)
	at org.apache.camel.component.file.GenericFileConsumer.doStart(GenericFileConsumer.java:640)
	at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
	at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:2042)
	at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2336)
	at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2272)
	at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:2202)
	at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1981)
	at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1851)
	at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1683)
	at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
	at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:1651)
	at org.apache.camel.component.servletlistener.CamelServletContextListener.contextInitialized(CamelServletContextListener.java:179)
	at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run(EventsManager.java:661)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
	at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
	at weblogic.servlet.internal.EventsManager.executeContextListener(EventsManager.java:241)
	at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:198)
	at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:183)
	at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1783)
	at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:2807)
	at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1661)
	at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:822)
	at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:360)
	at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:356)
	at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)
	at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:138)
	at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124)
	at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:213)
	at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:208)
	at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)
	at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:70)
	at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:24)
	at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:729)
	at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)
	at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:258)
	at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:61)
	at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:165)
	at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:80)
	at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:586)
	at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:148)
	at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:114)
	at weblogic.deploy.internal.targetserver.operations.StartOperation.doCommit(StartOperation.java:151)
	at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:339)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:846)
	at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1275)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:442)
	at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.commit(DeploymentServiceDispatcher.java:176)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doCommitCallback(DeploymentReceiverCallbackDeliverer.java:195)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$100(DeploymentReceiverCallbackDeliverer.java:13)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$2.run(DeploymentReceiverCallbackDeliverer.java:68)
	at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:295)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:254)

I have started working on a fix that adopt the approach for clustering found in QuartzEndpoint to QuartzSchedulerPollConsumerScheduler.

The patch is attached and I am asking for feedback.

Issue Links

• is related to: CAMEL-7663 SFTP consumers cannot be triggered by Quartz2 scheduler with JDBC trigger persistence: NotSerializableException
• requires: CAMEL-7663 SFTP consumers cannot be triggered by Quartz2 scheduler with JDBC trigger persistence: NotSerializableException
• links to: GitHub Pull Request #1577
• links to: GitHub Pull Request #1579