Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-3074

SolrPluginUtils.docListToSolrDocumentList is broken, existing test is also broken

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 3.5
    • 3.6, 4.0-ALPHA
    • search
    • None

    Description

      testDocListConversion() is not testing what it's suppossed to test. Because added test documents are not committed.

      http://search-lucene.com/m/uwh9l2SHH4e

      Attachments

        1. SOLR-3074.patch
          6 kB
          Chris M. Hostetter
        2. SOLR-3074.patch
          3 kB
          Chris M. Hostetter
        3. SOLR-3074.patch
          1 kB
          Ahmet Arslan

        Activity

          Ahmet: thanks for raising this issue, and for your patch.

          you didn't mention it in jira, but in your linked email you suggested that the method being tested (docListToSolrDocumentList) isn't actually doing what it's suppose to...

          Regardless of Set<String> fields parameter, SolrPluginUtils#docListToSolrDocumentList method loads all of the stored fields. Shouldn't it just load the fields given in the set? Should I file a jira ticket?

          Digging deeper, i found two problems...

          • docListToSolrDocumentList is delegating the field list to SolrIndexSearcher.doc, which means those fields are read immediately, but the rest of the fields will be lazy loaded
          • docListToSolrDocumentList is calling DocumentBuilder.loadStoredFields which loads all of the fields – regardless of which field list was passed to docListToSolrDocumentList
          • docListToSolrDocumentList has some really bizare code in it, preventing stored fields from being returned if they are the result of a copyField.

          Since there are no other usages of DocumentBuilder.loadStoredFields in Solr, it's not clear to me what the intended point of it's crazy logic was

          hossman Chris M. Hostetter added a comment - Ahmet: thanks for raising this issue, and for your patch. you didn't mention it in jira, but in your linked email you suggested that the method being tested (docListToSolrDocumentList) isn't actually doing what it's suppose to... Regardless of Set<String> fields parameter, SolrPluginUtils#docListToSolrDocumentList method loads all of the stored fields. Shouldn't it just load the fields given in the set? Should I file a jira ticket? Digging deeper, i found two problems... docListToSolrDocumentList is delegating the field list to SolrIndexSearcher.doc, which means those fields are read immediately, but the rest of the fields will be lazy loaded docListToSolrDocumentList is calling DocumentBuilder.loadStoredFields which loads all of the fields – regardless of which field list was passed to docListToSolrDocumentList docListToSolrDocumentList has some really bizare code in it, preventing stored fields from being returned if they are the result of a copyField. Since there are no other usages of DocumentBuilder.loadStoredFields in Solr, it's not clear to me what the intended point of it's crazy logic was

          additions to Ahmet's patch showing the additional problems found.

          given that neither of these two methods are used anywhere in Solr, and they are so clearly broken i suggest we deprecate them in 3.6 as "completely broken" and remove in trunk.

          hossman Chris M. Hostetter added a comment - additions to Ahmet's patch showing the additional problems found. given that neither of these two methods are used anywhere in Solr, and they are so clearly broken i suggest we deprecate them in 3.6 as "completely broken" and remove in trunk.

          barring objections i'm going to...

          • remove these methods in trunk
          • mark then "@deprecated apparently this method has behaved as documented for a very long time, and is unneeded in Solr, so it is deprecated and will be removed in Solr 4" in 3x

          ...tomorrow.

          hossman Chris M. Hostetter added a comment - barring objections i'm going to... remove these methods in trunk mark then "@deprecated apparently this method has behaved as documented for a very long time, and is unneeded in Solr, so it is deprecated and will be removed in Solr 4" in 3x ...tomorrow.
          ryantxu Ryan McKinley added a comment -

          remove these methods in trunk

          +1

          I think this was used long ago before the javabin format took over serializing DocList

          ryantxu Ryan McKinley added a comment - remove these methods in trunk +1 I think this was used long ago before the javabin format took over serializing DocList

          Oy...

          so i started down the road of removing all this from trunk, and then realized i somehow overlooked the fact that clustering is using docListToSolrDocumentList

          so then i started trying to fix clustering to have the logic it needed, and then realized that the code i wound up with actually implemented the documented contract of docListToSolrDocumentList in a way that made the new tests in this issue pass.

          So this new patch:

          • fixes docListToSolrDocumentList
          • fixes the test for docListToSolrDocumentList
          • removes DocumentBuilder.loadStoredFields

          my new plan is to commit as is to trunk, and backport all but the DocumentBuilder.java changes to 3x (i'll just deprecate loadStoredFields in 3x)

          comments?

          hossman Chris M. Hostetter added a comment - Oy... so i started down the road of removing all this from trunk, and then realized i somehow overlooked the fact that clustering is using docListToSolrDocumentList so then i started trying to fix clustering to have the logic it needed, and then realized that the code i wound up with actually implemented the documented contract of docListToSolrDocumentList in a way that made the new tests in this issue pass. So this new patch: fixes docListToSolrDocumentList fixes the test for docListToSolrDocumentList removes DocumentBuilder.loadStoredFields my new plan is to commit as is to trunk, and backport all but the DocumentBuilder.java changes to 3x (i'll just deprecate loadStoredFields in 3x) comments?

          r1303977 - trunk fix of SolrPluginUtils & test, removal of loadStoredFields method

          r1304001 - backport to 3x, but leave loadStoredFields in place with a scary freaking deprecation warning.

          Ahmet: thanks for your report and especially for your test case

          hossman Chris M. Hostetter added a comment - r1303977 - trunk fix of SolrPluginUtils & test, removal of loadStoredFields method r1304001 - backport to 3x, but leave loadStoredFields in place with a scary freaking deprecation warning. Ahmet: thanks for your report and especially for your test case

          People

            iorixxx Ahmet Arslan
            iorixxx Ahmet Arslan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: