Lucene - Core
  1. Lucene - Core
  2. LUCENE-3784 Switching tests infrastructure to randomizedtesting.*
  3. LUCENE-3808

Switch LuceneTestCaseRunner to RandomizedRunner. Enforce Random sharing contracts. Enforce thread leaks.

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: general/test
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

      Switch the runner to RandomizedRunner. Enforce the following:

      • Random sharing will result in a failure/ exception.
      • Add a validator for testXXX without @Test annotation. (custom test provider added).
      • Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
      • Add a validator for static hook shadowing (no-no).
      • Modify custom execution groups in LTC to be real @Groups.

      Follow up from merging with trunk:

      • rename tests.threadspercpu into tests.jvms (Steve)
      • align to 80 cols terminals (Robert) - use simple class names and add truncation padding.
      • the output is emitted twice in case of suite-level errors.

        Issue Links

          Activity

          Dawid Weiss created issue -
          Hide
          Robert Muir added a comment -

          Add a validator for testXXX without @Test annotation.

          Can we not require @Test? I think it should be fine to use testXXX without @Test. @Test is useless.

          Show
          Robert Muir added a comment - Add a validator for testXXX without @Test annotation. Can we not require @Test? I think it should be fine to use testXXX without @Test. @Test is useless.
          Hide
          Uwe Schindler added a comment -

          Dawid: Please let us name our tests like in JUnit3, our old test framework has a good test runner improvement for that, we don't like the verbosity on imports and this annotation.

          Show
          Uwe Schindler added a comment - Dawid: Please let us name our tests like in JUnit3, our old test framework has a good test runner improvement for that, we don't like the verbosity on imports and this annotation.
          Hide
          Erik Hatcher added a comment -

          Can we not require @Test? I think it should be fine to use testXXX without @Test. @Test is useless.

          +100000

          I was wondering if anyone would ever speak up about this. @Test is ridiculous. (so is @Override too, for that matter) - metadata is in the model implicitly.

          Show
          Erik Hatcher added a comment - Can we not require @Test? I think it should be fine to use testXXX without @Test. @Test is useless. +100000 I was wondering if anyone would ever speak up about this. @Test is ridiculous. (so is @Override too, for that matter) - metadata is in the model implicitly.
          Hide
          Dawid Weiss added a comment -

          Ok, ok, ok... I will not enforce @Test, don't kill me.

          Show
          Dawid Weiss added a comment - Ok, ok, ok... I will not enforce @Test, don't kill me.
          Dawid Weiss made changes -
          Field Original Value New Value
          Description Switch the runner to RandomizedRunner. Enforce the following:
          - Random sharing will result in a failure/ exception.
          - Thread leaks will result in a failure (add lingering if needed, but no ignores).
          - Add a validator for @Test method overrides (no-no).
          - Add a validator for static hook shadowing (no-no).
          - Add a validator for testXXX without @Test annotation.
          - Modify custom execution groups in LTC to be real @Groups.
          Switch the runner to RandomizedRunner. Enforce the following:
          - Random sharing will result in a failure/ exception.
          - Thread leaks will result in a failure (add lingering if needed, but no ignores).
          - Add a validator for @Test method overrides (no-no).
          - Add a validator for static hook shadowing (no-no).
          - -Add a validator for testXXX without @Test annotation.-
          - Modify custom execution groups in LTC to be real @Groups.
          Hide
          Uwe Schindler added a comment -

          Show
          Uwe Schindler added a comment -
          Hide
          Uwe Schindler added a comment -

          I was wondering if anyone would ever speak up about this. @Test is ridiculous. (so is @Override too, for that matter) - metadata is in the model implicitly.

          @Override is indeed not really different but makes sense when you refactor classes. The big PRO is: no import required as its in java.lang. Thats the most horrible thing on @Test

          Show
          Uwe Schindler added a comment - I was wondering if anyone would ever speak up about this. @Test is ridiculous. (so is @Override too, for that matter) - metadata is in the model implicitly. @Override is indeed not really different but makes sense when you refactor classes. The big PRO is: no import required as its in java.lang. Thats the most horrible thing on @Test
          Hide
          Robert Muir added a comment -

          @Override is indeed not really different but makes sense when you refactor classes.

          Interfaces too

          Show
          Robert Muir added a comment - @Override is indeed not really different but makes sense when you refactor classes. Interfaces too
          Hide
          Dawid Weiss added a comment -

          I like @Override, I even wish it were an explicit keyword like in C# (although I tend to like Java's by-default virtual methods more than C#'s explicit virtuals).

          Show
          Dawid Weiss added a comment - I like @Override, I even wish it were an explicit keyword like in C# (although I tend to like Java's by-default virtual methods more than C#'s explicit virtuals).
          Hide
          Dawid Weiss added a comment -

          I've hacked a local branch where I substituted LuceneTestCaseRunner for RandomizedRunner. And boy, hell broke loose

          So... the biggest issue I'm facing is indeed with Random sharing across threads. One may argue that sharing a Random instance across threads (with data races) is in fact making it even more random, but this doesn't seem to be the direction consistent with the possibility of consistently re-running the tests given the same starting seed. I know it isn't possible to coordinate threads anyway but I would still like to have Random instances to return identical sequences from within a single thread's perspective.

          After this lengthy introduction, here comes an example of Random sharing:

             > Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: This Random was created for/by another thread (Thread[RandomizedRunner-SuiteThread-org.apache.lucene.search.TestSearcherManager-seed#[3F1A26A6DA253628],5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager]). Random instances must not be shared (acquire per-thread). Current thread: Thread[Thread-53,5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager]
             > 	at __randomizedtesting.SeedInfo.seed([3F1A26A6DA253628]:0)
             > 	at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:307)
             > Caused by: java.lang.IllegalStateException: This Random was created for/by another thread (Thread[RandomizedRunner-SuiteThread-org.apache.lucene.search.TestSearcherManager-seed#[3F1A26A6DA253628],5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager]). Random instances must not be shared (acquire per-thread). Current thread: Thread[Thread-53,5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager]
             > 	at com.carrotsearch.randomizedtesting.RandomNoSetSeed.checkValid(RandomNoSetSeed.java:124)
             > 	at com.carrotsearch.randomizedtesting.RandomNoSetSeed.nextInt(RandomNoSetSeed.java:72)
             > 	at org.apache.lucene.util.LuceneTestCase.rarely(LuceneTestCase.java:938)
             > 	at org.apache.lucene.analysis.MockAnalyzer.maybePayload(MockAnalyzer.java:102)
             > 	at org.apache.lucene.analysis.MockAnalyzer.createComponents(MockAnalyzer.java:95)
             > 	at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:83)
             > 	at org.apache.lucene.document.Field.tokenStream(Field.java:467)
             > 	at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:86)
             > 	at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:287)
             > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:241)
             > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376)
             > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1533)
             > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1505)
             > 	at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase.updateDocument(ThreadedIndexingAndSearchingTestCase.java:111)
             > 	at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:259)
             > Caused by: com.carrotsearch.randomizedtesting.NotAnException: Original allocation stack for this Random.
             > 	at java.lang.Thread.getStackTrace(Thread.java:1479)
             > 	at com.carrotsearch.randomizedtesting.RandomNoSetSeed.<init>(RandomNoSetSeed.java:26)
             > 	at com.carrotsearch.randomizedtesting.Randomness.<init>(Randomness.java:20)
             > 	at com.carrotsearch.randomizedtesting.Randomness.<init>(Randomness.java:24)
             > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.run(RandomizedRunner.java:531)
             > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:551)
             > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$3(RandomizedRunner.java:495)
             > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:472)
          

          What's happening is that a Random is pushed to MockAnalyzer and then multiple threads access the same instance. I am investigating possible workarounds for this – Random instances are in fact thread local in RandomizedRunner so it would be possible to just grab your own Random in places like MockAnalyzer, without passing a ready to use instance.

          An ostrich solution is to dodge the issue and simply drop checking if a Random instance is shared or not, but it wouldn't be so much fun then, would it?

          Show
          Dawid Weiss added a comment - I've hacked a local branch where I substituted LuceneTestCaseRunner for RandomizedRunner. And boy, hell broke loose So... the biggest issue I'm facing is indeed with Random sharing across threads. One may argue that sharing a Random instance across threads (with data races) is in fact making it even more random, but this doesn't seem to be the direction consistent with the possibility of consistently re-running the tests given the same starting seed. I know it isn't possible to coordinate threads anyway but I would still like to have Random instances to return identical sequences from within a single thread's perspective. After this lengthy introduction, here comes an example of Random sharing: > Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: This Random was created for/by another thread (Thread[RandomizedRunner-SuiteThread-org.apache.lucene.search.TestSearcherManager-seed#[3F1A26A6DA253628],5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager]). Random instances must not be shared (acquire per-thread). Current thread: Thread[Thread-53,5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager] > at __randomizedtesting.SeedInfo.seed([3F1A26A6DA253628]:0) > at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:307) > Caused by: java.lang.IllegalStateException: This Random was created for/by another thread (Thread[RandomizedRunner-SuiteThread-org.apache.lucene.search.TestSearcherManager-seed#[3F1A26A6DA253628],5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager]). Random instances must not be shared (acquire per-thread). Current thread: Thread[Thread-53,5,RandomizedRunner-SuiteThreadGroup-org.apache.lucene.search.TestSearcherManager] > at com.carrotsearch.randomizedtesting.RandomNoSetSeed.checkValid(RandomNoSetSeed.java:124) > at com.carrotsearch.randomizedtesting.RandomNoSetSeed.nextInt(RandomNoSetSeed.java:72) > at org.apache.lucene.util.LuceneTestCase.rarely(LuceneTestCase.java:938) > at org.apache.lucene.analysis.MockAnalyzer.maybePayload(MockAnalyzer.java:102) > at org.apache.lucene.analysis.MockAnalyzer.createComponents(MockAnalyzer.java:95) > at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:83) > at org.apache.lucene.document.Field.tokenStream(Field.java:467) > at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:86) > at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:287) > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:241) > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376) > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1533) > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1505) > at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase.updateDocument(ThreadedIndexingAndSearchingTestCase.java:111) > at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:259) > Caused by: com.carrotsearch.randomizedtesting.NotAnException: Original allocation stack for this Random. > at java.lang.Thread.getStackTrace(Thread.java:1479) > at com.carrotsearch.randomizedtesting.RandomNoSetSeed.<init>(RandomNoSetSeed.java:26) > at com.carrotsearch.randomizedtesting.Randomness.<init>(Randomness.java:20) > at com.carrotsearch.randomizedtesting.Randomness.<init>(Randomness.java:24) > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.run(RandomizedRunner.java:531) > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:551) > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$3(RandomizedRunner.java:495) > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:472) What's happening is that a Random is pushed to MockAnalyzer and then multiple threads access the same instance. I am investigating possible workarounds for this – Random instances are in fact thread local in RandomizedRunner so it would be possible to just grab your own Random in places like MockAnalyzer, without passing a ready to use instance. An ostrich solution is to dodge the issue and simply drop checking if a Random instance is shared or not, but it wouldn't be so much fun then, would it?
          Hide
          Dawid Weiss added a comment -

          I've found this in the code:

              private final Random r;
              public MockIndexWriter(Random r, Directory dir, IndexWriterConfig conf) throws IOException {
                super(dir, conf);
                // must make a private random since our methods are
                // called from different threads; else test failures may
                // not be reproducible from the original seed
                this.r = new Random(r.nextInt());
          

          Ha!

          Show
          Dawid Weiss added a comment - I've found this in the code: private final Random r; public MockIndexWriter(Random r, Directory dir, IndexWriterConfig conf) throws IOException { super(dir, conf); // must make a private random since our methods are // called from different threads; else test failures may // not be reproducible from the original seed this.r = new Random(r.nextInt()); Ha!
          Dawid Weiss made changes -
          Description Switch the runner to RandomizedRunner. Enforce the following:
          - Random sharing will result in a failure/ exception.
          - Thread leaks will result in a failure (add lingering if needed, but no ignores).
          - Add a validator for @Test method overrides (no-no).
          - Add a validator for static hook shadowing (no-no).
          - -Add a validator for testXXX without @Test annotation.-
          - Modify custom execution groups in LTC to be real @Groups.
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - Thread leaks will result in a failure (add lingering if needed, but no ignores).
          - Add a validator for @Test method overrides (no-no).
          - Add a validator for static hook shadowing (no-no).
          - Modify custom execution groups in LTC to be real @Groups.
          Hide
          Michael McCandless added a comment -

          So... the biggest issue I'm facing is indeed with Random sharing across threads.

          I think we should always (try to!) make a thread-private, consistently seed'd Random instance...

          True, the scheduling of the threads by the JVM will always be a source of non-determinism, but we shouldn't make that worse. And, for some bugs, having consistent thread-private Random will make the bug fully reproducible (ie if the bug isn't a thread scheduling issue).

          Show
          Michael McCandless added a comment - So... the biggest issue I'm facing is indeed with Random sharing across threads. I think we should always (try to!) make a thread-private, consistently seed'd Random instance... True, the scheduling of the threads by the JVM will always be a source of non-determinism, but we shouldn't make that worse. And, for some bugs, having consistent thread-private Random will make the bug fully reproducible (ie if the bug isn't a thread scheduling issue).
          Hide
          Dawid Weiss added a comment -

          I'm working on this on my github branch. This is a good test case for the runner itself.

          Show
          Dawid Weiss added a comment - I'm working on this on my github branch. This is a good test case for the runner itself.
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - Thread leaks will result in a failure (add lingering if needed, but no ignores).
          - Add a validator for @Test method overrides (no-no).
          - Add a validator for static hook shadowing (no-no).
          - Modify custom execution groups in LTC to be real @Groups.
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - Thread leaks will result in a failure (add lingering if needed, but no ignores).
          - Add a validator for @Test method overrides (no-no).
          - Add a validator for static hook shadowing (no-no).
          - Modify custom execution groups in LTC to be real @Groups.
          Hide
          Dawid Weiss added a comment -

          I've fixed all offending tests under lucene/
          https://github.com/dweiss/lucene_solr/commit/1d9a7effd06cfc0b393b7e00b8b7f755bd967700

          All tests pass with RandomizedRunner there and the output from tests that failed was an excellent source to quickly get at the offending place.

          This looks very good to me so far. Solr and modules next.

          Show
          Dawid Weiss added a comment - I've fixed all offending tests under lucene/ https://github.com/dweiss/lucene_solr/commit/1d9a7effd06cfc0b393b7e00b8b7f755bd967700 All tests pass with RandomizedRunner there and the output from tests that failed was an excellent source to quickly get at the offending place. This looks very good to me so far. Solr and modules next.
          Dawid Weiss made changes -
          Link This issue incorporates LUCENE-3785 [ LUCENE-3785 ]
          Dawid Weiss committed 1296238 (1 file)
          Reviews: none

          A side effect of LUCENE-3808: validation of shadowed static hooks
          indicated this one as the violator. Since this is empty and cannot be
          overriden anyway I'm removing it.
          Removed shadowed empty afterClass() hook.

          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - Thread leaks will result in a failure (add lingering if needed, but no ignores).
          - Add a validator for @Test method overrides (no-no).
          - Add a validator for static hook shadowing (no-no).
          - Modify custom execution groups in LTC to be real @Groups.
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.
          - Thread leaks will result in a failure (add lingering if needed, but no ignores). [this is done, but disabled]
          - Add a validator for @Test method overrides (check how many of these we already have first).
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.
          - Thread leaks will result in a failure (add lingering if needed, but no ignores). [this is done, but disabled]
          - Add a validator for @Test method overrides (check how many of these we already have first).
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.
          - Thread leaks will result in a failure (add lingering if needed, but no ignores). [this is done, but disabled]
          - Add a validator for @Test method overrides (check how many of these we already have first).
          - What to do with thread-shared Random instances copies in MockIndexWriter and MockAnalyzer?
          Dawid Weiss made changes -
          Link This issue incorporates LUCENE-3429 [ LUCENE-3429 ]
          Hide
          Dawid Weiss added a comment -

          I'm planning to merge github branched code into trunk this weekend. It's been running in parallel for some time now on my build server and it seems to have the same failure coverage and at the same time is a start to clean up LuceneTestCase and associated test code.

          I hope you'll also like the new infrastructure – will elaborate about this a bit once merged.

          Show
          Dawid Weiss added a comment - I'm planning to merge github branched code into trunk this weekend. It's been running in parallel for some time now on my build server and it seems to have the same failure coverage and at the same time is a start to clean up LuceneTestCase and associated test code. I hope you'll also like the new infrastructure – will elaborate about this a bit once merged.
          Hide
          Dawid Weiss added a comment -

          Moving remaining bullets to separate issues.

          Show
          Dawid Weiss added a comment - Moving remaining bullets to separate issues.
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.
          - Thread leaks will result in a failure (add lingering if needed, but no ignores). [this is done, but disabled]
          - Add a validator for @Test method overrides (check how many of these we already have first).
          - What to do with thread-shared Random instances copies in MockIndexWriter and MockAnalyzer?
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.
          Dawid Weiss committed 1326351 (525 files)
          Reviews: none

          LUCENE-3808: Switch LuceneTestCaseRunner to RandomizedRunner. Enforce Random sharing contracts. Enforce thread leaks.

          Lucene trunk
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert)
          Dawid Weiss committed 1326406 (2 files)
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert)
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert)
          - the output is emitted twice in case of suite-level errors.
          Dawid Weiss made changes -
          Link This issue incorporates LUCENE-3988 [ LUCENE-3988 ]
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert)
          - the output is emitted twice in case of suite-level errors.
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert) - use simple class names and add truncation padding.
          - the output is emitted twice in case of suite-level errors.
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert) - use simple class names and add truncation padding.
          - the output is emitted twice in case of suite-level errors.
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - (/) rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert) - use simple class names and add truncation padding.
          - the output is emitted twice in case of suite-level errors.
          sarowe committed 1326466 (1 file)
          Reviews: none

          LUCENE-3808: Maven configuration: The lucene-core module needs randomizedtesting dependency independently from the test-framework module, since under maven, lucene-core includes test-framework source instead of depending on the test-framework module

          Hide
          Robert Muir added a comment -

          minor spelling nit:

          [junit4] Ignoring seed attribute because it is overriden by user properties.

          overriden should be overridden...

          Show
          Robert Muir added a comment - minor spelling nit: [junit4] Ignoring seed attribute because it is overriden by user properties. overriden should be overridden...
          Hide
          Dawid Weiss added a comment -

          This wasn't a typo, I had it wrong all over the place. Corrected, thanks for pointing this out.

          Show
          Dawid Weiss added a comment - This wasn't a typo, I had it wrong all over the place. Corrected, thanks for pointing this out.
          Dawid Weiss made changes -
          Description Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - (/) rename tests.threadspercpu into tests.jvms (Steve)
          - align to 80 cols terminals (Robert) - use simple class names and add truncation padding.
          - the output is emitted twice in case of suite-level errors.
          Dev. branch at: https://github.com/dweiss/lucene_solr/tree/rr

          Switch the runner to RandomizedRunner. Enforce the following:
          - (/) Random sharing will result in a failure/ exception.
          - (/) -Add a validator for testXXX without @Test annotation.- (custom test provider added).
          - (/) Make sure tests are executed with assertions enabled (at least for solr/lucene packages).
          - (/) Add a validator for static hook shadowing (no-no).
          - (/) Modify custom execution groups in LTC to be real @Groups.

          Follow up from merging with trunk:
          - (/) rename tests.threadspercpu into tests.jvms (Steve)
          - (/) align to 80 cols terminals (Robert) - use simple class names and add truncation padding.
          - (/) the output is emitted twice in case of suite-level errors.
          Dawid Weiss made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Dawid Weiss committed 1326570 (1 file)
          Reviews: none

          LUCENE-3808: added changes entry.

          Dawid Weiss committed 1326689 (1 file)
          Reviews: none

          LUCENE-3808: configuration of terminal output for tests. I think I found a
          way to please everybody after all... I've defined a set of properties
          which one can override using local properties (read before anything else).
          The defaults are full class names, no truncating. If one wants short names
          (or ellipsis or no output for suites or no output for a particular type of
          status, whatever) then define any of these properties in your local
          configuration file:

            <!-- Override these in your local properties to your desire. -->
            <!-- Show simple class names (no package) in test suites. -->
            <property name="tests.useSimpleNames" value="false" />
            <!-- Max width for class name truncation. -->
            <property name="tests.maxClassNameColumns" value="10000" />
            <!-- Show suite summaries for tests. -->
            <property name="tests.showSuiteSummary" value="true" />
            <!-- Configure test emission to console for each type of status -->
            <property name="tests.showSuccess" value="false" />
            <property name="tests.showError" value="true" />
            <property name="tests.showFailure" value="true" />
            <property name="tests.showIgnored" value="true" />

          sarowe committed 1326780 (1 file)
          Reviews: none

          LUCENE-3808: Segregate RandomizedTesting library sysprops from other Lucene/Solr tests.* sysprops, and use the empty string value to tell the test runner to use the default value for these sysprops.

          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Dawid Weiss
              Reporter:
              Dawid Weiss
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development