[CAMEL-13722] Quartz job paused, but not resumed after zookeeper connection recovery leads to job misfire

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Bug
• Affects Version/s: 2.22.0
• Fix Version/s: Future
• Component/s: camel-quartz2
• Estimated Complexity: Unknown
• Assignee: Unassigned
• Reporter: Camel
• Created: 04/Jul/19 08:18
• Updated: 18/May/20 19:59
• Resolved: 18/May/20 19:59

Description

We have 2 instances of route coordinated by zookeeper with MasterRoutePolicy. If zookeeper is disconnected, the quartz job is paused and left in that state after connection recovery. Later on, if another node shuts down, current node with paused job starts the route, despite wrong time. 

How to reproduce:

1. Run 2 instances with MasterRoutePolicy for leader selection connected to zookeeper with route from("quartz2://foobar?cron=0+0+15+?+*+MON-FRI").log("start");
2. Simulate zookeeper disconnect (pull net cable etc.) on non-master -> Trigger is set to Paused. Log:
   [Curator-ConnectionStateManager-0] org.apache.camel.component.quartz2.QuartzEndpoint: Pausing trigger foobar
3.  Reconnect to zookeeper
4.  Master node successfully runs the scheduled job, wait 5 min 
5.  Shutdown master node -> the second node triggers the job even though it was already executed on master and cron time is in the past

Expected behavior is to skip running missed job in step 5.