[OAK-2725] Wrong indexed query estimates exceed more than double the actual index entries

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 1.1.8
• Component/s: query
• Assignee: Unassigned
• Reporter: Florin Iordache
• Created: 07/Apr/15 13:57
• Updated: 08/Oct/19 15:22
• Resolved: 08/Apr/15 08:33

Description

The ApproximateCounter.adjustCountSync public method that is used by the indexing engine will sometimes produce very unrealistic cost estimates.

The problem is that it can produce an estimated cost that exceeds the estimated cost of the full traversal query, thus causing the index to be bypassed altogether, resulting in a full traversal rather than the use of the existing index.

Problem resides in the way the property counts are updated:

• The count property update goes through if two randoms are equal to zero: random(100) and random( {1, 2, 4, 8, 16, ...}

  ).

• Same static pseudo random generator for all invocations.

Even if #1 might seem improbable, it is statistically possible to reach a very high count with only a handful of invocations.

In practice I've found that running 100 tests with 1000 invocations if the adjustCountSync method will yield costs exceeding value 2000 in 4-10% of the tests. Attaching a patch for ApproximateCounterTest with this test case.

Issue Links

• is related to: OAK-1907 Better cost estimates for traversal, property, and ordered indexes