Details
Minor Description
Summary
We encountered an infinite loop while downloading files using the SFTP endpoint.
During our investigation, we discovered the InProgressRepository utilizing the SimpleLRUCache. This cache incorporates an unconventional eviction implementation that involves a changes tracking queue.
Any attempt to modify the cache, even when putting the same element, results in an increase in the cache size, potentially causing eviction to occur even for a single element.
The code is as follows:
Map<Object, Object> lruCache = LRUCacheFactory.newLRUCache(1, 1); int numberOfInserts = 0; lruCache.put("key", "value"); while (lruCache.size() > 0) { lruCache.put("key", "value"); numberOfInserts++; } throw new IllegalStateException("We should never reach this point, we inserted the same element " + numberOfInserts + " times");
In the provided code, the failure occurs when the number of inserts reaches 2.
Expected Result
Utilize the LRU Cache limited only by the maximum size passed into it, rather than by some internal implementation limits.
Proposed Solution
Implement a standard LRUCache, say, by leveraging a LRUMap from Apache Commons.
We may provide the PR later on.
The Background
We need to use a dynamic SFTP route (the download parameters depend on some configuration passed from the outside). By default, this is impossible, so we will have to use pollEnrich:
.pollEnrich().simple(SFTP_DYNAMIC_URI)
.timeout(RETURN_CURRENT_RESULT_WITHOUT_WAIT)
The pollEnrich takes only one element (see GenericFilePollingConsumer). The file consumer requests from the SFTP endpoint the list of all files, then takes the first, then gets the list again and takes the second file, and so on.
The combination of InProgressRepository and idempotent consumer is used to avoid handling the same files again.
However, every time the SFTP endpoint lists files, the endpoints and polling consumers adjust the list in the InProgressRepository, thus affecting the SimpleLRUCache.
The number of elements in the cache doesn't grow, however, the number of changes grows with almost n^2 speed.
With the limit of 50k files, the SimpleLRUCache is evicted already on ~600
files. This leads to an infinite loop while processing the files.
Attachments
Issue Links
- is related to
-
CAMEL-20039 camel-core - SimpleLRUCache add support for soft cache
-
- Resolved
-
- links to