Solr
  1. Solr
  2. SOLR-3074

SolrPluginUtils.docListToSolrDocumentList is broken, existing test is also broken

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.5
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: search
    • Labels:
      None

      Description

      testDocListConversion() is not testing what it's suppossed to test. Because added test documents are not committed.

      http://search-lucene.com/m/uwh9l2SHH4e

      1. SOLR-3074.patch
        6 kB
        Hoss Man
      2. SOLR-3074.patch
        3 kB
        Hoss Man
      3. SOLR-3074.patch
        1 kB
        Ahmet Arslan

        Activity

        Hide
        Hoss Man added a comment -

        Ahmet: thanks for raising this issue, and for your patch.

        you didn't mention it in jira, but in your linked email you suggested that the method being tested (docListToSolrDocumentList) isn't actually doing what it's suppose to...

        Regardless of Set<String> fields parameter, SolrPluginUtils#docListToSolrDocumentList method loads all of the stored fields. Shouldn't it just load the fields given in the set? Should I file a jira ticket?

        Digging deeper, i found two problems...

        • docListToSolrDocumentList is delegating the field list to SolrIndexSearcher.doc, which means those fields are read immediately, but the rest of the fields will be lazy loaded
        • docListToSolrDocumentList is calling DocumentBuilder.loadStoredFields which loads all of the fields – regardless of which field list was passed to docListToSolrDocumentList
        • docListToSolrDocumentList has some really bizare code in it, preventing stored fields from being returned if they are the result of a copyField.

        Since there are no other usages of DocumentBuilder.loadStoredFields in Solr, it's not clear to me what the intended point of it's crazy logic was

        Show
        Hoss Man added a comment - Ahmet: thanks for raising this issue, and for your patch. you didn't mention it in jira, but in your linked email you suggested that the method being tested (docListToSolrDocumentList) isn't actually doing what it's suppose to... Regardless of Set<String> fields parameter, SolrPluginUtils#docListToSolrDocumentList method loads all of the stored fields. Shouldn't it just load the fields given in the set? Should I file a jira ticket? Digging deeper, i found two problems... docListToSolrDocumentList is delegating the field list to SolrIndexSearcher.doc, which means those fields are read immediately, but the rest of the fields will be lazy loaded docListToSolrDocumentList is calling DocumentBuilder.loadStoredFields which loads all of the fields – regardless of which field list was passed to docListToSolrDocumentList docListToSolrDocumentList has some really bizare code in it, preventing stored fields from being returned if they are the result of a copyField. Since there are no other usages of DocumentBuilder.loadStoredFields in Solr, it's not clear to me what the intended point of it's crazy logic was
        Hide
        Hoss Man added a comment -

        additions to Ahmet's patch showing the additional problems found.

        given that neither of these two methods are used anywhere in Solr, and they are so clearly broken i suggest we deprecate them in 3.6 as "completely broken" and remove in trunk.

        Show
        Hoss Man added a comment - additions to Ahmet's patch showing the additional problems found. given that neither of these two methods are used anywhere in Solr, and they are so clearly broken i suggest we deprecate them in 3.6 as "completely broken" and remove in trunk.
        Hide
        Hoss Man added a comment -

        barring objections i'm going to...

        • remove these methods in trunk
        • mark then "@deprecated apparently this method has behaved as documented for a very long time, and is unneeded in Solr, so it is deprecated and will be removed in Solr 4" in 3x

        ...tomorrow.

        Show
        Hoss Man added a comment - barring objections i'm going to... remove these methods in trunk mark then "@deprecated apparently this method has behaved as documented for a very long time, and is unneeded in Solr, so it is deprecated and will be removed in Solr 4" in 3x ...tomorrow.
        Hide
        Ryan McKinley added a comment -

        remove these methods in trunk

        +1

        I think this was used long ago before the javabin format took over serializing DocList

        Show
        Ryan McKinley added a comment - remove these methods in trunk +1 I think this was used long ago before the javabin format took over serializing DocList
        Hide
        Hoss Man added a comment -

        Oy...

        so i started down the road of removing all this from trunk, and then realized i somehow overlooked the fact that clustering is using docListToSolrDocumentList

        so then i started trying to fix clustering to have the logic it needed, and then realized that the code i wound up with actually implemented the documented contract of docListToSolrDocumentList in a way that made the new tests in this issue pass.

        So this new patch:

        • fixes docListToSolrDocumentList
        • fixes the test for docListToSolrDocumentList
        • removes DocumentBuilder.loadStoredFields

        my new plan is to commit as is to trunk, and backport all but the DocumentBuilder.java changes to 3x (i'll just deprecate loadStoredFields in 3x)

        comments?

        Show
        Hoss Man added a comment - Oy... so i started down the road of removing all this from trunk, and then realized i somehow overlooked the fact that clustering is using docListToSolrDocumentList so then i started trying to fix clustering to have the logic it needed, and then realized that the code i wound up with actually implemented the documented contract of docListToSolrDocumentList in a way that made the new tests in this issue pass. So this new patch: fixes docListToSolrDocumentList fixes the test for docListToSolrDocumentList removes DocumentBuilder.loadStoredFields my new plan is to commit as is to trunk, and backport all but the DocumentBuilder.java changes to 3x (i'll just deprecate loadStoredFields in 3x) comments?
        Hide
        Hoss Man added a comment -

        r1303977 - trunk fix of SolrPluginUtils & test, removal of loadStoredFields method

        r1304001 - backport to 3x, but leave loadStoredFields in place with a scary freaking deprecation warning.

        Ahmet: thanks for your report and especially for your test case

        Show
        Hoss Man added a comment - r1303977 - trunk fix of SolrPluginUtils & test, removal of loadStoredFields method r1304001 - backport to 3x, but leave loadStoredFields in place with a scary freaking deprecation warning. Ahmet: thanks for your report and especially for your test case

          People

          • Assignee:
            Ahmet Arslan
            Reporter:
            Ahmet Arslan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development