Description
Queries should be as fast as possible:
- They should read as little data as possible (low I/O)
- Network roundtrips should be reduced (see also
OAK-9780) - In-memory processing should be fast (low CPU usage)
To do that:
- Queries needs to have the right indexes. Possibly indexes need to be added (which might be a manual task, or semi-automated, or fully automated). For a developer, it would also be good to know how fast a query could be, if an index is added.
- Queries should use the right indexes. Sometimes multiple indexes can be used.
- Queries should use the right execution plan (for example: a join can be evaluated in multiple ways).
For this, it is great to have accurate statistics. We currently have statistics about number of nodes per path (approximate counter), and document statistics for Lucene and Elastic indexes.
But we don't have statistics for unindexed data currently. That would be good to have: which property (by property name) is how common? How many distinct values are there per property? What is the histogram? And so on. For this, something like the counter index could be added, that is updated using a streaming algorithm. We need to ensure the number of writes to this index is low (e.g. less than 1% of the overall writes), and memory usage is very low. There are a number of such libraries, but arguably we could implement this ourselves, as our use case is untypical (reduced number of writes, reduced memory usage). https://github.com/thomasmueller/tinyStats and related libraries could be used as a starting point.
Additional use cases:
- Property names that are only used once or twice are likely typos
Attachments
Issue Links
- relates to
-
OAK-9780 Prefetch node states
-
- Closed
-