[OAK-3001] Simplify JournalGarbageCollector using a dedicated timestamp property

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.14, 1.6.0
• Component/s: core, mongomk
• Labels: scalability
• Epic Link: DocumentMK scalability improvements
• Assignee: Vikas Saurabh
• Reporter: Stefan Egli
• Created: 17/Jun/15 06:56
• Updated: 26/Sep/19 11:43
• Resolved: 18/Nov/16 16:12

Description

This subtask is about spawning out a comment from Chetan Mehrotra re JournalGC:

Further looking at JournalGarbageCollector ... it would be simpler if you record the journal entry timestamp as an attribute in JournalEntry document and then you can delete all the entries which are older than some time by a simple query. This would avoid fetching all the entries to be deleted on the Oak side

and a corresponding reply from myself:

Re querying by timestamp: that would indeed be simpler. With the current set of DocumentStore API however, I believe this is not possible. But: DocumentStore.query comes quite close: it would probably just require the opposite of that method too:

    public <T extends Document> List<T> query(Collection<T> collection,
                                              String fromKey,
                                              String toKey,
                                              String indexedProperty,
                                              long endValue,
                                              int limit) {

.. or what about generalizing this method to have both a startValue and an endValue - with -1 indicating when one of them is not used?

Issue Links

• is blocked by: OAK-3213 Improve DocumentStore API
• is blocked by: OAK-3085 Add timestamp property to journal entries
• relates to: OAK-2829 Comparing node states for external changes is too slow
• relates to: OAK-3975 Change DELETE_BATCH_SIZE to 100 (from 450), and make it configurable
• relates to: OAK-3976 journal should support large(r) entries

Sub-Tasks

• DocumentStore: add method to remove with a condition on an indexed property: Closed
• JournalGarbageCollector: use new DocumentStore remove() method: Closed
• RDBDocumentStore: implement new conditional remove method: Closed
• MongoDocumentStore: implement new conditional remove method: Closed
• JournalGarbageCollector: journalGcBatchSize configuration is irrelevant post implementation tasks of bulk remove support in DocumentStore: Closed