Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
unomi-2.0.0, unomi-1.6.1
-
None
Description
The purge system have been fixed recently in version 1.6.0: https://jira.jahia.org/browse/UNOMI-504
The fix that was done, just fixed the query used for identifying the profiles to be purged.
Actually we figured a bit late that the purge system contains more issues than that.
Here is a non exhaustive list of the one already identified and that will need fix/answers an even reflections/discussions for some of them
Bug 1: disabling the profile purge will end up with all profiles to be purge everyday
Using this conf:
org.apache.unomi.profile.purge.existTime = -1 org.apache.unomi.profile.purge.inactiveTime = -1 org.apache.unomi.monthly.index.purge.existTime= 12
To only enable events/session purge will have a bad side effect: all profiles of the system will be removed everyday.
bug 2: OOM issue for profile purge.
The persistence service is implementing a method:
/**
* Deletes items with the specified Item subclass matching the specified {@link Condition}.
*
* @param <T> the type of the Item subclass we want to delete
* @param query a {@link Condition} identifying which elements we want to delete
* @param clazz the {@link Item} subclass of the items we want to delete
* @return {@code true} if the deletion was successful, {@code false} otherwise
*/
<T extends Item> boolean removeByQuery(Condition query, Class<T> clazz);
The issue with the implem is that it's doing the following:
- query profiles to get the IDs of documents
- building a batch request with a delete request per profile ID
- sending the batch.
So in case of a lot profiles to be purge at once, the batch can grow a lot, and consume all the memory available.
We should not do like that, instead we should:
- just use the ES query: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html directly with the query to identify the profiles
Bug 3: What is the cache system implemented in the persistence service.
Seem's coming from: https://issues.apache.org/jira/browse/UNOMI-166
It's used by the persistence service, and it explain why the removeByQuery implem is bad.
The reason why the removeByQuery is querying all the documents IDs is to invalidate the cache.
But it's strange, the cache system is not used at all:
<cm:property name="itemClassesToCache" value="" />
So I would say, we should remove this cache system, because:
- it's not tested
- nobody know what would be the effect if enabled
- it's not possible to activate it with ENV conf, or global unomi conf.
- it's not documented
- it's impacting implems like: removeByQuery, even if the cache system is not used
- it's mostly dead code inside the core persistence of Unomi, witch is dangerous to keep.
Update during team planning: This caching system should be removed.
Bug 4: Purge job seem's never unregistered.
We do register the purge job when the profile service is started, but in case the profile service is stopped (stop the bundle for example). Then it's not unregistered.
Bug 5: what should we do in case of unomi cluster, should all of them do the purge ?
In case of unomi cluster, all of them may try to trigger the purge, I dont think it's an issue, but it look like something could be done to avoid doing unecessary operations twice or more.
