Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10072

The test TestSelectiveWeightCreation appears to be unreliable.

    Details

    • Type: Test
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, master (7.0)
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      TestSelectiveWeightCreation 17.00% unreliable 30.00 24.66

      1. stdout
        61 kB
        Mark Miller
      2. stdout
        61 kB
        Mark Miller

        Issue Links

          Activity

          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I beasted tests at 12 at a time, but 10 may be a more reasonable number as 12 seems to really start pushing RAM exhaustion. I'll rerun this test at 10 at a time and post fail logs.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I beasted tests at 12 at a time, but 10 may be a more reasonable number as 12 seems to really start pushing RAM exhaustion. I'll rerun this test at 10 at a time and post fail logs.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I got 2 fails out of the 30, both are attached.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I got 2 fails out of the 30, both are attached.
          Hide
          cpoerschke Christine Poerschke added a comment -

          Thanks Mark. When you say "12 at a time" or "10 at a time", how can I specify the same, to try and reproduce locally, before and after any changes?

          Show
          cpoerschke Christine Poerschke added a comment - Thanks Mark. When you say "12 at a time" or "10 at a time", how can I specify the same, to try and reproduce locally, before and after any changes?
          Hide
          markrmiller@gmail.com Mark Miller added a comment - - edited

          I use this beasting script: https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/78d1f5c97021455ddb0eb2b3ba93b44589089244/gistfile1.sh

          I end up basically doing:

          cd lucene-solr/solr
          bash beast.sh -i 30 -p 10 TestSelectiveWeightCreation

          Each JVM can use 512MB of RAM though, so 10 can be RAM hungry. It can also be overkill if you don't have enough processors. I use those numbers for a machine with 16 cores.

          I believe the built in ant based beast target will also allow you to specify how many times to run in parallel, but I'm not familiar with it currently.

          Show
          markrmiller@gmail.com Mark Miller added a comment - - edited I use this beasting script: https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/78d1f5c97021455ddb0eb2b3ba93b44589089244/gistfile1.sh I end up basically doing: cd lucene-solr/solr bash beast.sh -i 30 -p 10 TestSelectiveWeightCreation Each JVM can use 512MB of RAM though, so 10 can be RAM hungry. It can also be overkill if you don't have enough processors. I use those numbers for a machine with 16 cores. I believe the built in ant based beast target will also allow you to specify how many times to run in parallel, but I'm not familiar with it currently.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Though, actually, I've recently learned it is much more efficient to move into the module with the test:

          cd lucene-solr/solr/contrib/ltr
          bash beast.sh -i 30 -p 10 TestSelectiveWeightCreation

          Show
          markrmiller@gmail.com Mark Miller added a comment - Though, actually, I've recently learned it is much more efficient to move into the module with the test: cd lucene-solr/solr/contrib/ltr bash beast.sh -i 30 -p 10 TestSelectiveWeightCreation
          Hide
          cpoerschke Christine Poerschke added a comment -

          Thanks. From delving into lucene/common-build.xml I think the ant equivalent then would be

          ant beast -Dbeast.iters=30 -Dtests.dups=10 -Dtestcase=TestSelectiveWeightCreation
          
          Show
          cpoerschke Christine Poerschke added a comment - Thanks. From delving into lucene/common-build.xml I think the ant equivalent then would be ant beast -Dbeast.iters=30 -Dtests.dups=10 -Dtestcase=TestSelectiveWeightCreation
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Yeah, looks like that makes sense.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Yeah, looks like that makes sense.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user mnilsson23 opened a pull request:

          https://github.com/apache/lucene-solr/pull/156

          SOLR-10072: Fix test reliability

          There is no guarantee to order of docs with the same score

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/mnilsson23/lucene-solr SOLR-10072

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/156.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #156


          commit 1368570a8d0c1460d0b8285ca287d27ba74d8f59
          Author: mnilsson23 <mnilsson23@bloomberg.net>
          Date: 2017-02-10T20:01:29Z

          SOLR-10072: Fix test reliability
          There is no guarantee to order of docs with same score


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user mnilsson23 opened a pull request: https://github.com/apache/lucene-solr/pull/156 SOLR-10072 : Fix test reliability There is no guarantee to order of docs with the same score You can merge this pull request into a Git repository by running: $ git pull https://github.com/mnilsson23/lucene-solr SOLR-10072 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/156.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #156 commit 1368570a8d0c1460d0b8285ca287d27ba74d8f59 Author: mnilsson23 <mnilsson23@bloomberg.net> Date: 2017-02-10T20:01:29Z SOLR-10072 : Fix test reliability There is no guarantee to order of docs with same score
          Hide
          mnilsson Michael Nilsson added a comment -

          I opened up a PR to master that should fix the reliability of the test. Since there is no guarantee to the order of documents which have the same score, I made sure that the ids being checked in the test are now using features which will make the scores different.

          Show
          mnilsson Michael Nilsson added a comment - I opened up a PR to master that should fix the reliability of the test. Since there is no guarantee to the order of documents which have the same score, I made sure that the ids being checked in the test are now using features which will make the scores different.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Thanks! I'll test it out.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Thanks! I'll test it out.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          This test is solid now with this patch applied, thanks!

          Show
          markrmiller@gmail.com Mark Miller added a comment - This test is solid now with this patch applied, thanks!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 387c8e9c2daff15bb1cba09aeffc5d9e5723cd4e in lucene-solr's branch refs/heads/master from markrmiller
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=387c8e9 ]

          SOLR-10072: The test TestSelectiveWeightCreation appears to be unreliable.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 387c8e9c2daff15bb1cba09aeffc5d9e5723cd4e in lucene-solr's branch refs/heads/master from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=387c8e9 ] SOLR-10072 : The test TestSelectiveWeightCreation appears to be unreliable.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a2bd720e67ea930459dabc5e49f4d44d03e0042b in lucene-solr's branch refs/heads/branch_6x from markrmiller
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a2bd720 ]

          SOLR-10072: The test TestSelectiveWeightCreation appears to be unreliable.

          Show
          jira-bot ASF subversion and git services added a comment - Commit a2bd720e67ea930459dabc5e49f4d44d03e0042b in lucene-solr's branch refs/heads/branch_6x from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a2bd720 ] SOLR-10072 : The test TestSelectiveWeightCreation appears to be unreliable.

            People

            • Assignee:
              markrmiller@gmail.com Mark Miller
              Reporter:
              markrmiller@gmail.com Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development