Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10159

DBQ, where query is based on updated value, reordered with the update doesn't work

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Background/History

      If a recently updated (in-place) value is used for DBQ, the DBQ doesn't work at Lucene level, unless there's an explicit commit between the update and the DBQ, due to LUCENE-7344. To work around this, Yonik suggested that we use ulog.openRealtimeSearcher() just before the DBQ is performed. This worked fine.

      Example:

      ADD: [id=0, dv=200, title="mytitle", _version_=100]
      UPD: [id=0, dv=300, _version_=200]
      DBQ: q="dv:300", _version_=300
      

      Problem discovered now

      Suppose, in the above example, the last two commands are reordered at the replica. What would happen is: (i) the full document (_version_ 100) is received and indexed, (ii) the DBQ is received (out of ordered) and applied, and no document is deleted [so far so good] and this DBQ is buffered in ulog.deleteByQueries map, (iii) the in-place update arrives (_version 200), it is applied to the document that was added in step i. After that, the buffered DBQ is applied (at DUH2.addAndDelete()). This buffered DBQ, based on a value updated immediately before (step ii), fails to delete the document.

      What happens exactly?

      The initial DBQ query is "dv:300", but when it is applied, it is expanded to "+dv:[300 TO 300] -ConstantScore(frange(long(_version_)):[300 TO *])". In spite of doing a ulog.openRealtimeSearcher() just before the DBQ, it doesn't work.

      A different version of the query, i.e. "+dv:[300 TO 300] +_version_:[200 TO 200]" also doesn't work. As I found out, this happened due to the presence of two clauses! "+dv:[300 TO 300]" works, and so does "+_version_:[200 TO 200]", but both clauses don't work together. Also, surprisingly, even "+dv:[300 TO 300] +dv:[300 TO 300]" doesn't work (same clause repeated).

      Investigation at Lucene level

      Upon some tedious investigation into the internals of Lucene, I discovered that if I change the internal search (at BufferedUpdates) to use Sort.RELEVANCE instead of Sort.INDEXORDER (which, I think is the default when using weight/scorer), the DBQ is applied correctly.

        Attachments

        1. SOLR-10159.patch
          6 kB
          Ishan Chattopadhyaya
        2. SOLR-10159.patch
          5 kB
          Ishan Chattopadhyaya

          Issue Links

            Activity

              People

              • Assignee:
                ichattopadhyaya Ishan Chattopadhyaya
                Reporter:
                ichattopadhyaya Ishan Chattopadhyaya
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: