[SOLR-17296] rerank w/scaling (still) broken when using debug to get explain info

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 9.4
• Fix Version/s: main (10.0), 9.7, 9.6.1
• Assignee: Chris M. Hostetter
• Reporter: Chris M. Hostetter
• Created: 16/May/24 19:19
• Updated: 29/May/24 23:52
• Resolved: 17/May/24 23:36

Description

The changes made in SOLR-16931 (9.4) attempted to work around problems that existed when attempting to enable degugging (to get score explanations) in combination with using reRankScale ...

The reason for this is that in order to do proper explain for minMaxScaling you need to know the min and max score in the result set. This piece of state is maintained in the ReRankScaler itself which is inside of the ReRankQuery. But for this information to be populated the query must first be run. In distributed mode, explain is called in the second pass when the ids query is run so the state needed for the explain is not populated. ...

However, the solution attempted was incomplete and failed to account for multiple factors...

... The PR attached to this addresses this problem by doing a single pass distributed query if debugQuery is turned on and if reRank score scaling is applied. I'll add a distributed test for this as well.

This change is very limited in scope because the single pass distributed is only switched on in the very specific case when debugQuery=true and reRankScaling is on.

 

• NPEs are still possible...
  • Instead of checking for ResponseBuilder.isDebugResults() (which is what triggers explain logic) the new code only checked for specific debug request param combinations:
    • debuQuery=true (a legacy option intended only for backcompat)

• • • debug=true (intended as an alias for debug=all
  • It did not check for either of these options, which if used will still trigger an NPE...
    • debug=results (which actually dictates the value of ResponseBuilder.isDebugResults()
    • debug=all (a short hand for setting all debug options)
• the attempt to force a single pass query didn't modify the correct variable
  • The new code modified a conditional based on a boolean distribSinglePass for setting sreq.purpose and rb.onePassDistributedQuery
  • But it did not modify the value of the boolean distribSinglePass itself - meaning other logic that uses that variable in that method still assumes multiple passes will be used.
  • In particular, these means that even though a single pass is used for both PURPOSE_GET_TOP_IDS and PURPOSE_GET_FIELDS the full "fl" requested by the user is not propagated as part of this request
    • Only the uniqueKey and any sot fields are ultimately returned to the user.

Issue Links

• is broken by: SOLR-16931 ReRankScaler explain breaks with debug=true and in distributed mode
• split to: SOLR-17299 Redesign debug component / reranking to support multi-stage debugging
• links to: GitHub Pull Request #2465