[OAK-6180] Tune cursor batch/limit size

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.3, 1.8.0
• Component/s: mongomk
• Epic Link: Improve DocumentMK performance
• Assignee: Marcel Reutegger
• Reporter: Marcel Reutegger
• Created: 08/May/17 07:51
• Updated: 06/Jul/17 15:34
• Resolved: 03/Jul/17 07:12

Description

MongoDocumentStore uses the default batch size, which means MongoDB will initially get 100 documents and then as many documents as fit into 4MB. Depending on the document size, the number of documents may be quite high and the risk of running into the 60 seconds query timeout defined by Oak increases.

Tuning the batch size (or using a limit) may also be helpful in optimizing the amount of data transferred from MongoDB to Oak. The DocumentNodeStore fetches child nodes in batches as well. The logic there is slightly different. The initial batch size is 100 and every subsequent batch doubles in size until it reaches 1600. Bandwidth is wasted if the MongoDB Java driver fetches way more than requested by Oak.