Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: 2.16.2
-
Component/s: camel-core, camel-quartz2
-
Labels:None
-
Environment:
ALL
-
Patch Info:Patch Available
-
Estimated Complexity:Novice
-
Flags:Patch
Description
When any file:// or ftp:// consumer has a quartz2 schedule it can start throwing exceptions because too many worker-threads are busy at the same time (FileNotFoundException).
Both workers can find the same files during filling the maxMessagesPerPoll, or during processing of those files. This happens when the route can not process all files before the next trigger happens.
example (every minute):
from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter=#fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler.cron=0+0/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerId=nl_bij-export-to-archive-276)
to(file:///data/work/sl/work-archive/work/276/)
Attached you can find a stacktrace that would happen very often if worker1 processes and moves files that worker2 would also like to start processing.
This does not happen when using scheduler=spring or when using delay=1m.
The only way I have found to make sure a file or ftp component does not use multiple threads while consuming very large batches is to annotate the QuartzScheduledPollConsumerJob class with @DisallowConcurrentExecution.
I am not familiar enough with the Camelcode to say what side effects it has and if this would prevent any quartz job in camel to now be single threaded, even if the user does not want it to be.
But to me it looks like an oversight when moving from quartz to quartz2. A file or ftp consumer should be single threaded while retrieving.