[SOLR-10159] DBQ, where query is based on updated value, reordered with the update doesn't work

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.5, 7.0
• Assignee: Ishan Chattopadhyaya
• Reporter: Ishan Chattopadhyaya
• Created: 18/Feb/17 20:23
• Updated: 08/Jun/19 15:32
• Resolved: 19/Feb/17 08:44

Description

Background/History

If a recently updated (in-place) value is used for DBQ, the DBQ doesn't work at Lucene level, unless there's an explicit commit between the update and the DBQ, due to LUCENE-7344. To work around this, Yonik suggested that we use ulog.openRealtimeSearcher() just before the DBQ is performed. This worked fine.

Example:

ADD: [id=0, dv=200, title="mytitle", _version_=100]
UPD: [id=0, dv=300, _version_=200]
DBQ: q="dv:300", _version_=300

Problem discovered now

Suppose, in the above example, the last two commands are reordered at the replica. What would happen is: (i) the full document (_version_ 100) is received and indexed, (ii) the DBQ is received (out of ordered) and applied, and no document is deleted [so far so good] and this DBQ is buffered in ulog.deleteByQueries map, (iii) the in-place update arrives (_version 200), it is applied to the document that was added in step i. After that, the buffered DBQ is applied (at DUH2.addAndDelete()). This buffered DBQ, based on a value updated immediately before (step ii), fails to delete the document.

What happens exactly?

The initial DBQ query is "dv:300", but when it is applied, it is expanded to "+dv:[300 TO 300] -ConstantScore(frange(long(_version_)):[300 TO *])". In spite of doing a ulog.openRealtimeSearcher() just before the DBQ, it doesn't work.

A different version of the query, i.e. "+dv:[300 TO 300] +_version_:[200 TO 200]" also doesn't work. As I found out, this happened due to the presence of two clauses! "+dv:[300 TO 300]" works, and so does "+_version_:[200 TO 200]", but both clauses don't work together. Also, surprisingly, even "+dv:[300 TO 300] +dv:[300 TO 300]" doesn't work (same clause repeated).

Investigation at Lucene level

Upon some tedious investigation into the internals of Lucene, I discovered that if I change the internal search (at BufferedUpdates) to use Sort.RELEVANCE instead of Sort.INDEXORDER (which, I think is the default when using weight/scorer), the DBQ is applied correctly.

Issue Links

• is related to: LUCENE-7344 Deletion by query of uncommitted docs not working with DV updates
• is related to: SOLR-5944 Support updates of numeric DocValues