Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9288

RTG: fl=[docid] silently missing for uncommitted docs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.2, master (7.0)
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Found in SOLR-9180 testing.

      when using RTG in a single node solr install, the [docid] transformer works for committed docs, but is silently missing from uncommited docs.

      this inconsistency is confusing. It seems like even if there is no valid docid to return in this case, the key should still be present in the resulting doc.

      I would suggest using either null or -1 in this case?

        Issue Links

          Activity

          Hide
          hossman Hoss Man added a comment -

          See TestPseudoReturnFields.testDocIdAugmenterRTG for example...

              // behavior shouldn't matter if we are committed or uncommitted
              for (String id : Arrays.asList("42","99")) {
                assertQ(id + ": fl=[docid]",
                        req("qt","/get","id",id, "wt","xml", "fl","[docid]")
                        ,"count(//doc)=1"
                        ,"//doc/int[@name='[docid]']"
                        ,"//doc[count(*)=1]"
                        );
              }
          

          current failure...

             [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPseudoReturnFields -Dtests.method=testDocIdAugmenterRTG -Dtests.seed=98335D83793D2329 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Pacific/Enderbury -Dtests.asserts=true -Dtests.file.encoding=UTF-8
             [junit4] ERROR   0.08s J1 | TestPseudoReturnFields.testDocIdAugmenterRTG <<<
             [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
             [junit4]    > 	at __randomizedtesting.SeedInfo.seed([98335D83793D2329:74889C9A69FA53]:0)
             [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:780)
             [junit4]    > 	at org.apache.solr.search.TestPseudoReturnFields.testDocIdAugmenterRTG(TestPseudoReturnFields.java:554)
             [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
             [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//doc/int[@name='[docid]']
             [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
             [junit4]    > <response>
             [junit4]    > <doc name="doc"></doc>
             [junit4]    > </response>
             [junit4]    > 	request was:qt=/get&fl=[docid]&id=99&wt=xml
             [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:773)
          

          NOTE: whatever solution is decided on, once the inconsistency is resolved, there are a lot of other test methods in TestPseudoReturnFields that can be updated to also exercise [docid]

          Show
          hossman Hoss Man added a comment - See TestPseudoReturnFields.testDocIdAugmenterRTG for example... // behavior shouldn't matter if we are committed or uncommitted for ( String id : Arrays.asList( "42" , "99" )) { assertQ(id + ": fl=[docid]" , req( "qt" , "/get" , "id" ,id, "wt" , "xml" , "fl" , "[docid]" ) , "count( //doc)=1" , " //doc/ int [@name='[docid]']" , " //doc[count(*)=1]" ); } current failure... [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestPseudoReturnFields -Dtests.method=testDocIdAugmenterRTG -Dtests.seed=98335D83793D2329 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Pacific/Enderbury -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 0.08s J1 | TestPseudoReturnFields.testDocIdAugmenterRTG <<< [junit4] > Throwable #1: java.lang.RuntimeException: Exception during query [junit4] > at __randomizedtesting.SeedInfo.seed([98335D83793D2329:74889C9A69FA53]:0) [junit4] > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:780) [junit4] > at org.apache.solr.search.TestPseudoReturnFields.testDocIdAugmenterRTG(TestPseudoReturnFields.java:554) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//doc/int[@name='[docid]'] [junit4] > xml response was: <?xml version="1.0" encoding="UTF-8"?> [junit4] > <response> [junit4] > <doc name="doc"></doc> [junit4] > </response> [junit4] > request was:qt=/get&fl=[docid]&id=99&wt=xml [junit4] > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:773) NOTE: whatever solution is decided on, once the inconsistency is resolved, there are a lot of other test methods in TestPseudoReturnFields that can be updated to also exercise [docid]
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit fee9526208375fec6a7651249b182fbca1a29703 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fee9526 ]

          SOLR-9180: More comprehensive tests of psuedo-fields for RTG and SolrCloud requests

          This commit also includes new @AwaitsFix'ed tests for the following known issues...

          • SOLR-9285 ArrayIndexOutOfBoundsException when ValueSourceAugmenter used with RTG on uncommitted doc
          • SOLR-9286 SolrCloud RTG: psuedo-fields (like ValueSourceAugmenter, [shard], etc...) silently fails (even for committed doc)
          • SOLR-9287 single node RTG: NPE if score is requested
          • SOLR-9288 RTG: fl=[docid] silently missing for uncommitted docs
          • SOLR-9289 SolrCloud RTG: fl=[docid] silently ignored for all docs

          (cherry picked from commit ae316f1e39e58d89758f997913a38059d74ccb47)

          Show
          jira-bot ASF subversion and git services added a comment - Commit fee9526208375fec6a7651249b182fbca1a29703 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fee9526 ] SOLR-9180 : More comprehensive tests of psuedo-fields for RTG and SolrCloud requests This commit also includes new @AwaitsFix'ed tests for the following known issues... SOLR-9285 ArrayIndexOutOfBoundsException when ValueSourceAugmenter used with RTG on uncommitted doc SOLR-9286 SolrCloud RTG: psuedo-fields (like ValueSourceAugmenter, [shard] , etc...) silently fails (even for committed doc) SOLR-9287 single node RTG: NPE if score is requested SOLR-9288 RTG: fl= [docid] silently missing for uncommitted docs SOLR-9289 SolrCloud RTG: fl= [docid] silently ignored for all docs (cherry picked from commit ae316f1e39e58d89758f997913a38059d74ccb47)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ae316f1e39e58d89758f997913a38059d74ccb47 in lucene-solr's branch refs/heads/master from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ae316f1 ]

          SOLR-9180: More comprehensive tests of psuedo-fields for RTG and SolrCloud requests

          This commit also includes new @AwaitsFix'ed tests for the following known issues...

          • SOLR-9285 ArrayIndexOutOfBoundsException when ValueSourceAugmenter used with RTG on uncommitted doc
          • SOLR-9286 SolrCloud RTG: psuedo-fields (like ValueSourceAugmenter, [shard], etc...) silently fails (even for committed doc)
          • SOLR-9287 single node RTG: NPE if score is requested
          • SOLR-9288 RTG: fl=[docid] silently missing for uncommitted docs
          • SOLR-9289 SolrCloud RTG: fl=[docid] silently ignored for all docs
          Show
          jira-bot ASF subversion and git services added a comment - Commit ae316f1e39e58d89758f997913a38059d74ccb47 in lucene-solr's branch refs/heads/master from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ae316f1 ] SOLR-9180 : More comprehensive tests of psuedo-fields for RTG and SolrCloud requests This commit also includes new @AwaitsFix'ed tests for the following known issues... SOLR-9285 ArrayIndexOutOfBoundsException when ValueSourceAugmenter used with RTG on uncommitted doc SOLR-9286 SolrCloud RTG: psuedo-fields (like ValueSourceAugmenter, [shard] , etc...) silently fails (even for committed doc) SOLR-9287 single node RTG: NPE if score is requested SOLR-9288 RTG: fl= [docid] silently missing for uncommitted docs SOLR-9289 SolrCloud RTG: fl= [docid] silently ignored for all docs
          Hide
          hossman Hoss Man added a comment -

          Ok, so the root cause seems to be this bit of code in DocIdAugmenterFactory...

            public void transform(SolrDocument doc, int docid, float score) {
              if( docid >= 0 ) {
                doc.setField( name, docid );
              }
          

          At first glance that looks like a conscious choice to not include a docid in the response for uncommitted docs – except that according to the logs that bit of code was added by SOLR-1566 and pre-dates the existence of the transaction log and RTG.


          I think we should go ahead and update this transformer return -1 for uncommitted docs, so people can reliably look for the key in the response and use the value to know if they are looking at committed/uncommited values, rather then determine that based on the presense of the key in the response.

          Show
          hossman Hoss Man added a comment - Ok, so the root cause seems to be this bit of code in DocIdAugmenterFactory... public void transform(SolrDocument doc, int docid, float score) { if ( docid >= 0 ) { doc.setField( name, docid ); } At first glance that looks like a conscious choice to not include a docid in the response for uncommitted docs – except that according to the logs that bit of code was added by SOLR-1566 and pre-dates the existence of the transaction log and RTG. I think we should go ahead and update this transformer return -1 for uncommitted docs, so people can reliably look for the key in the response and use the value to know if they are looking at committed/uncommited values, rather then determine that based on the presense of the key in the response.
          Hide
          hossman Hoss Man added a comment -

          proposed changes with tests

          Will try to commit soon unless I here objections.

          Show
          hossman Hoss Man added a comment - proposed changes with tests Will try to commit soon unless I here objections.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9feeee4e2764add63afab8f5b3784274f300a94f in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9feeee4 ]

          SOLR-9288: Fix [docid] transformer to return -1 when used in RTG with uncommitted doc

          (cherry picked from commit 08019f42889a537764384429c4184515d233a2cb)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9feeee4e2764add63afab8f5b3784274f300a94f in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9feeee4 ] SOLR-9288 : Fix [docid] transformer to return -1 when used in RTG with uncommitted doc (cherry picked from commit 08019f42889a537764384429c4184515d233a2cb)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 08019f42889a537764384429c4184515d233a2cb in lucene-solr's branch refs/heads/master from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=08019f4 ]

          SOLR-9288: Fix [docid] transformer to return -1 when used in RTG with uncommitted doc

          Show
          jira-bot ASF subversion and git services added a comment - Commit 08019f42889a537764384429c4184515d233a2cb in lucene-solr's branch refs/heads/master from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=08019f4 ] SOLR-9288 : Fix [docid] transformer to return -1 when used in RTG with uncommitted doc
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 08019f42889a537764384429c4184515d233a2cb in lucene-solr's branch refs/heads/apiv2 from Chris Hostetter
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=08019f4 ]

          SOLR-9288: Fix [docid] transformer to return -1 when used in RTG with uncommitted doc

          Show
          jira-bot ASF subversion and git services added a comment - Commit 08019f42889a537764384429c4184515d233a2cb in lucene-solr's branch refs/heads/apiv2 from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=08019f4 ] SOLR-9288 : Fix [docid] transformer to return -1 when used in RTG with uncommitted doc
          Hide
          mikemccand Michael McCandless added a comment -

          Bulk close resolved issues after 6.2.0 release.

          Show
          mikemccand Michael McCandless added a comment - Bulk close resolved issues after 6.2.0 release.

            People

            • Assignee:
              hossman Hoss Man
              Reporter:
              hossman Hoss Man
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development