Solr
  1. Solr
  2. SOLR-1915

DebugComponent should use NamedList to output Explanations instead of Explanation.toString()

    Details

      Description

      DebugComponent currently uses Explanation.toString() to "format" score explanations for each document as plain text with whitespace indenting to denote the hierarchical relationship, and then adds those explanations to the SolrQueryResponse.

      Instead DebugComponent should transform the Explanation objects into NamedLists so that the full structure can be formatted in a logical way by the ResponseWriter

        Activity

        Hide
        Hoss Man added a comment -

        Fun bit of trivia: this code actually existed in pre-apache Solar (i think it was about 10 lines) but because of a (short lived) bug in XmlResponseWriter that caused it to break horribly if it walked a nested data structure more then 8 levels deep, I yanked it back out.

        I can't believe it's taken me this long to think it would be handy to add it back in.

        Show
        Hoss Man added a comment - Fun bit of trivia: this code actually existed in pre-apache Solar (i think it was about 10 lines) but because of a (short lived) bug in XmlResponseWriter that caused it to break horribly if it walked a nested data structure more then 8 levels deep, I yanked it back out. I can't believe it's taken me this long to think it would be handy to add it back in.
        Hide
        Hoss Man added a comment -

        patch making the necessary changes.

        Just in case some poor soul was actually attempting to programmaticality parse the old "toString()" response, i added a new "debug.explain.stringFormat=true" param to force the old behavior, but the default is to be a structured response, and frankly i don't think we should even document that param except in the upgrade section of CHANGES.txt (where i explicitly indicated that it would be removed in the next release – it should only be there as a red flag and a stop gap in case people don't notice the change until it breaks something)

        Show
        Hoss Man added a comment - patch making the necessary changes. Just in case some poor soul was actually attempting to programmaticality parse the old "toString()" response, i added a new "debug.explain.stringFormat=true" param to force the old behavior, but the default is to be a structured response, and frankly i don't think we should even document that param except in the upgrade section of CHANGES.txt (where i explicitly indicated that it would be removed in the next release – it should only be there as a red flag and a stop gap in case people don't notice the change until it breaks something)
        Hide
        Yonik Seeley added a comment -

        How's the readability in the browser (after pressing CTRL-U of course, or when using something like JSON format)?
        If the more common usecase is human inspection, perhaps the string format should be the default?

        Show
        Yonik Seeley added a comment - How's the readability in the browser (after pressing CTRL-U of course, or when using something like JSON format)? If the more common usecase is human inspection, perhaps the string format should be the default?
        Hide
        Hoss Man added a comment -

        It's much more readable in the browser because you don't have to hit Ctrl-U to view the source – it's readable as is. it's also fairly readable in JSON (assuming you use indent=true, which is pretty much necessary to read any of the JSON output sanely)

        Show
        Hoss Man added a comment - It's much more readable in the browser because you don't have to hit Ctrl-U to view the source – it's readable as is. it's also fairly readable in JSON (assuming you use indent=true, which is pretty much necessary to read any of the JSON output sanely)
        Hide
        Hoss Man added a comment -

        Committed revision 949318.

        people can try this out on the trunk – if there are concerns about the readability in the browser, we can reopen and give the param a more user friendly name/default value

        Show
        Hoss Man added a comment - Committed revision 949318. people can try this out on the trunk – if there are concerns about the readability in the browser, we can reopen and give the param a more user friendly name/default value
        Hide
        Yonik Seeley added a comment -

        On a quick try, I think I like the old behavior as a default.
        http://localhost:8983/solr/select?fl=id,score&defType=edismax&qf=text&pf=text&q=solr%20server&debugQuery=on&debug.explain.stringFormat=true&indent=true

        This is what you get when you cut-n-paste the explain from the XML, or press CTRL-U to view the raw XML.

        0.4444791 = (MATCH) sum of:
          0.4444791 = (MATCH) sum of:
            0.22223955 = (MATCH) weight(text:solr in 0), product of:
              0.40824828 = queryWeight(text:solr), product of:
                3.0794415 = idf(docFreq=2, maxDocs=24)
                0.13257217 = queryNorm
              0.5443735 = (MATCH) fieldWeight(text:solr in 0), product of:
                1.4142135 = tf(termFreq(text:solr)=2)
                3.0794415 = idf(docFreq=2, maxDocs=24)
                0.125 = fieldNorm(field=text, doc=0)
            0.22223955 = (MATCH) weight(text:server in 0), product of:
              0.40824828 = queryWeight(text:server), product of:
                3.0794415 = idf(docFreq=2, maxDocs=24)
                0.13257217 = queryNorm
              0.5443735 = (MATCH) fieldWeight(text:server in 0), product of:
                1.4142135 = tf(termFreq(text:server)=2)
                3.0794415 = idf(docFreq=2, maxDocs=24)
                0.125 = fieldNorm(field=text, doc=0)
        

        And now this is the parser friendly output:
        http://localhost:8983/solr/select?fl=id,score&defType=edismax&qf=text&pf=text&q=solr%20server&debugQuery=on&debug.explain.stringFormat=false&indent=true

        This is also after pressing CTRL-U, because you can't really cut-n-paste from firefox from the XML view (it doesn't preserve any indenting).

          <lst name="SOLR1000">
        	<bool name="match">true</bool>
        	<float name="value">0.4444791</float>
        	<str name="description">sum of:</str>
        	<arr name="details">
        	 <lst>
        	  <bool name="match">true</bool>
        	  <float name="value">0.4444791</float>
        	  <str name="description">sum of:</str>
        	  <arr name="details">
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.22223955</float>
        		<str name="description">weight(text:solr in 0), product of:</str>
        		<arr name="details">
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.40824828</float>
        		<str name="description">queryWeight(text:solr), product of:</str>
        		<arr name="details">
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">3.0794415</float>
        		<str name="description">idf(docFreq=2, maxDocs=24)</str>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.13257217</float>
        		<str name="description">queryNorm</str>
        		</lst>
        		</arr>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.5443735</float>
        		<str name="description">fieldWeight(text:solr in 0), product of:</str>
        		<arr name="details">
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">1.4142135</float>
        		<str name="description">tf(termFreq(text:solr)=2)</str>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">3.0794415</float>
        		<str name="description">idf(docFreq=2, maxDocs=24)</str>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.125</float>
        		<str name="description">fieldNorm(field=text, doc=0)</str>
        		</lst>
        		</arr>
        		</lst>
        		</arr>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.22223955</float>
        		<str name="description">weight(text:server in 0), product of:</str>
        		<arr name="details">
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.40824828</float>
        		<str name="description">queryWeight(text:server), product of:</str>
        		<arr name="details">
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">3.0794415</float>
        		<str name="description">idf(docFreq=2, maxDocs=24)</str>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.13257217</float>
        		<str name="description">queryNorm</str>
        		</lst>
        		</arr>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.5443735</float>
        		<str name="description">fieldWeight(text:server in 0), product of:</str>
        		<arr name="details">
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">1.4142135</float>
        		<str name="description">tf(termFreq(text:server)=2)</str>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">3.0794415</float>
        		<str name="description">idf(docFreq=2, maxDocs=24)</str>
        		</lst>
        		<lst>
        		<bool name="match">true</bool>
        		<float name="value">0.125</float>
        		<str name="description">fieldNorm(field=text, doc=0)</str>
        		</lst>
        		</arr>
        		</lst>
        		</arr>
        		</lst>
        	  </arr>
        	 </lst>
        	</arr>
          </lst>
         </lst>
        

        I don't think I'd ever try to make use of the latter format for debugging (and I wouldn't want anyone sending that to the list), so it seems like the default should be the old behavior?

        Having firefox not show the linebreaks unless you press CTRL-U (or cut-n-paste it) also has it's advantages... the large explains take up less vertical room in the response in those cases where you're not looking at them.

        Show
        Yonik Seeley added a comment - On a quick try, I think I like the old behavior as a default. http://localhost:8983/solr/select?fl=id,score&defType=edismax&qf=text&pf=text&q=solr%20server&debugQuery=on&debug.explain.stringFormat=true&indent=true This is what you get when you cut-n-paste the explain from the XML, or press CTRL-U to view the raw XML. 0.4444791 = (MATCH) sum of: 0.4444791 = (MATCH) sum of: 0.22223955 = (MATCH) weight(text:solr in 0), product of: 0.40824828 = queryWeight(text:solr), product of: 3.0794415 = idf(docFreq=2, maxDocs=24) 0.13257217 = queryNorm 0.5443735 = (MATCH) fieldWeight(text:solr in 0), product of: 1.4142135 = tf(termFreq(text:solr)=2) 3.0794415 = idf(docFreq=2, maxDocs=24) 0.125 = fieldNorm(field=text, doc=0) 0.22223955 = (MATCH) weight(text:server in 0), product of: 0.40824828 = queryWeight(text:server), product of: 3.0794415 = idf(docFreq=2, maxDocs=24) 0.13257217 = queryNorm 0.5443735 = (MATCH) fieldWeight(text:server in 0), product of: 1.4142135 = tf(termFreq(text:server)=2) 3.0794415 = idf(docFreq=2, maxDocs=24) 0.125 = fieldNorm(field=text, doc=0) And now this is the parser friendly output: http://localhost:8983/solr/select?fl=id,score&defType=edismax&qf=text&pf=text&q=solr%20server&debugQuery=on&debug.explain.stringFormat=false&indent=true This is also after pressing CTRL-U, because you can't really cut-n-paste from firefox from the XML view (it doesn't preserve any indenting). <lst name= "SOLR1000" > <bool name= "match" > true </bool> < float name= "value" >0.4444791</ float > <str name= "description" >sum of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >0.4444791</ float > <str name= "description" >sum of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >0.22223955</ float > <str name= "description" >weight(text:solr in 0), product of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >0.40824828</ float > <str name= "description" >queryWeight(text:solr), product of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >3.0794415</ float > <str name= "description" >idf(docFreq=2, maxDocs=24)</str> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >0.13257217</ float > <str name= "description" >queryNorm</str> </lst> </arr> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >0.5443735</ float > <str name= "description" >fieldWeight(text:solr in 0), product of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >1.4142135</ float > <str name= "description" >tf(termFreq(text:solr)=2)</str> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >3.0794415</ float > <str name= "description" >idf(docFreq=2, maxDocs=24)</str> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >0.125</ float > <str name= "description" >fieldNorm(field=text, doc=0)</str> </lst> </arr> </lst> </arr> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >0.22223955</ float > <str name= "description" >weight(text:server in 0), product of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >0.40824828</ float > <str name= "description" >queryWeight(text:server), product of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >3.0794415</ float > <str name= "description" >idf(docFreq=2, maxDocs=24)</str> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >0.13257217</ float > <str name= "description" >queryNorm</str> </lst> </arr> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >0.5443735</ float > <str name= "description" >fieldWeight(text:server in 0), product of:</str> <arr name= "details" > <lst> <bool name= "match" > true </bool> < float name= "value" >1.4142135</ float > <str name= "description" >tf(termFreq(text:server)=2)</str> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >3.0794415</ float > <str name= "description" >idf(docFreq=2, maxDocs=24)</str> </lst> <lst> <bool name= "match" > true </bool> < float name= "value" >0.125</ float > <str name= "description" >fieldNorm(field=text, doc=0)</str> </lst> </arr> </lst> </arr> </lst> </arr> </lst> </arr> </lst> </lst> I don't think I'd ever try to make use of the latter format for debugging (and I wouldn't want anyone sending that to the list), so it seems like the default should be the old behavior? Having firefox not show the linebreaks unless you press CTRL-U (or cut-n-paste it) also has it's advantages... the large explains take up less vertical room in the response in those cases where you're not looking at them.
        Hide
        Hoss Man added a comment -

        bq Having firefox not show the linebreaks unless you press CTRL-U (or cut-n-paste it) also has it's advantages... the large explains take up less vertical room in the response in those cases where you're not looking at them.

        Hmm ... i suppose, but the ability to easily collapse things in the XML view by clicking the +/- seems so much nicer by default – i seriously find the XML view soooo much easier to read/use then the plain text.

        But you've got enough legitimate points that i'm totally on board with changing the default back to plain text – which also means we should come up with a better name for the option then "debug.explain.stringFormat=false"

        suggestions?

        Show
        Hoss Man added a comment - bq Having firefox not show the linebreaks unless you press CTRL-U (or cut-n-paste it) also has it's advantages... the large explains take up less vertical room in the response in those cases where you're not looking at them. Hmm ... i suppose, but the ability to easily collapse things in the XML view by clicking the +/- seems so much nicer by default – i seriously find the XML view soooo much easier to read/use then the plain text. But you've got enough legitimate points that i'm totally on board with changing the default back to plain text – which also means we should come up with a better name for the option then "debug.explain.stringFormat=false" suggestions?
        Hide
        Erik Hatcher added a comment -

        debug.explain.structured=true ??

        Show
        Erik Hatcher added a comment - debug.explain.structured=true ??
        Hide
        Hoss Man added a comment -

        yeah ... "debug.explain.structured" is along the lines of what i had in mind.

        Committed revision 950667.

        Show
        Hoss Man added a comment - yeah ... "debug.explain.structured" is along the lines of what i had in mind. Committed revision 950667.
        Hide
        Hoss Man added a comment -

        backported to 3x....
        Committed revision 1125535.

        Show
        Hoss Man added a comment - backported to 3x.... Committed revision 1125535.
        Hide
        Robert Muir added a comment -

        Bulk close for 3.2

        Show
        Robert Muir added a comment - Bulk close for 3.2

          People

          • Assignee:
            Hoss Man
            Reporter:
            Hoss Man
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development