Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9977

reproducible failures in DistribDocExpirationUpdateProcessorTest due to IndexWriterConfig randomization

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Here's an example of a seed that currently fails reliably on master...

      ant test  -Dtestcase=DistribDocExpirationUpdateProcessorTest -Dtests.method=test -Dtests.seed=34988FCF7C369D9 -Dtests.slow=true -Dtests.locale=el -Dtests.timezone=Etc/GMT+10 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
         [junit4]    > Throwable #1: java.lang.AssertionError: Exactly one shard should have changed, instead: [core_node1, core_node2] nodes=([expiry_shard2_replica1(core_node1), expiry_shard1_replica1(core_node2)]) expected:<1> but was:<2>
         [junit4]    > 	at __randomizedtesting.SeedInfo.seed([34988FCF7C369D9:8B1DB726593F0421]:0)
         [junit4]    > 	at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.test(DistribDocExpirationUpdateProcessorTest.java:116)
      

      The meat of the test is to verify that the periodic DBQs triggered by the DocExpirationUpdateProcessor don't cause unnecessary new searchers w/cache flushing/warming. – only shards affected by deltheetes get their searchers re-opened.

      enabling infoStream logging on the test shows that (something I havne't fully dug into in) the randomized IndexWriterConfig is causing the IndexWriter to generate a new segments file after a commit early in the test – completely unrelated to the DBQ+commit logic we're paying close attention to – that still contains the exact same underlying segments/docs. It's just a new segments file name with a new index version# – which throws off the index version# tracking the test is using to make sure only the segment that should be impacted by our DBQ is impacted by the DBQ.


      Since this kind of randomized index changing under the covers contradicts the methodology used in the test, it should be removed so we can reliably know that the only reason an reader/searcher changes is either because the solr code being tested does it deliberately, or because of a bug that needs fixed.

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 70f312943e53e60c27e3ccf589bd3829b63e5d84 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=70f3129 ]

        SOLR-9977: Fix config bug in DistribDocExpirationUpdateProcessorTest that allowed false assumptions about when index version changes

        (cherry picked from commit 9ee48aa857e15461dd6ec6482194141da72e0ba2)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 70f312943e53e60c27e3ccf589bd3829b63e5d84 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=70f3129 ] SOLR-9977 : Fix config bug in DistribDocExpirationUpdateProcessorTest that allowed false assumptions about when index version changes (cherry picked from commit 9ee48aa857e15461dd6ec6482194141da72e0ba2)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9ee48aa857e15461dd6ec6482194141da72e0ba2 in lucene-solr's branch refs/heads/master from Chris Hostetter
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9ee48aa ]

        SOLR-9977: Fix config bug in DistribDocExpirationUpdateProcessorTest that allowed false assumptions about when index version changes

        Show
        jira-bot ASF subversion and git services added a comment - Commit 9ee48aa857e15461dd6ec6482194141da72e0ba2 in lucene-solr's branch refs/heads/master from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9ee48aa ] SOLR-9977 : Fix config bug in DistribDocExpirationUpdateProcessorTest that allowed false assumptions about when index version changes
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit c58eac13378f618532190348574d96a72ef413e7 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c58eac1 ]

        SOLR-9977, SOLR-9979: move CHANGES entries to correct section (Bug)

        Aparently these were shifted during merge/cherry-picks

        Show
        jira-bot ASF subversion and git services added a comment - Commit c58eac13378f618532190348574d96a72ef413e7 in lucene-solr's branch refs/heads/branch_6x from Chris Hostetter [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c58eac1 ] SOLR-9977 , SOLR-9979 : move CHANGES entries to correct section (Bug) Aparently these were shifted during merge/cherry-picks

          People

          • Assignee:
            hossman Hoss Man
            Reporter:
            hossman Hoss Man
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development