Lucene - Core
  1. Lucene - Core
  2. LUCENE-3492

Extract a generic framework for running randomized tests.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: general/test
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The work on this issue is temporarily at github (lots of experiments and tweaking):
      https://github.com/carrotsearch/randomizedtesting
      Or directly: git clone git://github.com/carrotsearch/randomizedtesting.git


      RandomizedRunner is a JUnit runner, so it is capable of running @Test-annotated test cases. It
      respects regular lifecycle hooks such as @Before, @After, @BeforeClass or @AfterClass, but it
      also adds the following:

      Randomized, but repeatable execution and infrastructure for dealing with randomness:

      • uses pseudo-randomness (so that a given run can be repeated if given the same starting seed)
        for many things called "random" below,
      • randomly shuffles test methods to ensure they don't depend on each other,
      • randomly shuffles hooks (within a given class) to ensure they don't depend on each other,
      • base class RandomizedTest provides a number of methods for generating random numbers, strings
        and picking random objects from collections (again, this is fully repeatable given the
        initial seed if there are no race conditions),
      • the runner provides infrastructure to augment stack traces with information about the initial
        seeds used for running the test, so that it can be repeated (or it can be determined that
        the test is not repeatable – this indicates a problem with the test case itself).

      Thread control:

      • any threads created as part of a test case are assigned the same initial random seed
        (repeatability),
      • tracks and attempts to terminate any Threads that are created and not terminated inside
        a test case (not cleaning up causes a test failure),
      • tracks and attempts to terminate test cases that run for too long (default timeout: 60 seconds,
        adjustable using global property or annotations),

      Improved validation and lifecycle support:

      • RandomizedRunner uses relaxed contracts of hook methods' accessibility (hook methods can be
        private). This helps in avoiding problems with method shadowing (static hooks) or overrides
        that require tedious super.() chaining). Private hooks are always executed and don't affect
        subclasses in any way, period.
      • @Listeners annotation on a test class allows it to hook into the execution progress and listen
        to events.
      • @Validators annotation allows a test class to provide custom validation strategies
        (project-specific). For example a base class can request specific test case naming strategy
        (or reject JUnit3-like methods, for instance).
      • RandomizedRunner does not "chain" or "suppress" exceptions happening during execution of
        a test case (including hooks). All exceptions are reported as soon as they happened and multiple
        failure reports can occur. Most environments we know of then display these failures sequentially
        allowing a clearer understanding of what actually happened first.

        Issue Links

          Activity

          Dawid Weiss made changes -
          Link This issue relates to LUCENE-3784 [ LUCENE-3784 ]
          Dawid Weiss made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 4.0 [ 12314025 ]
          Resolution Fixed [ 1 ]
          Dawid Weiss made changes -
          Description I love the idea of randomized testing. Everyone (we at CarrotSearch, Lucene and Solr folks) have their glue to make it possible. The question is if there's something to pull out that others could share without having the need to import Lucene-specific classes.

          The work on this issue is on my github account (lots of experiments):
          https://github.com/dweiss/randomizedtesting

          Or directly: git clone git://github.com/dweiss/randomizedtesting.git
          {color:red}The work on this issue is temporarily at github{color} (lots of experiments and tweaking):
          https://github.com/carrotsearch/randomizedtesting
          Or directly: git clone git://github.com/carrotsearch/randomizedtesting.git
          {color}
          ----

          RandomizedRunner is a JUnit runner, so it is capable of running @Test-annotated test cases. It
          respects regular lifecycle hooks such as @Before, @After, @BeforeClass or @AfterClass, but it
          also adds the following:

          Randomized, but repeatable execution and infrastructure for dealing with randomness:

          - uses pseudo-randomness (so that a given run can be repeated if given the same starting seed)
            for many things called "random" below,
          - randomly shuffles test methods to ensure they don't depend on each other,
          - randomly shuffles hooks (within a given class) to ensure they don't depend on each other,
          - base class RandomizedTest provides a number of methods for generating random numbers, strings
            and picking random objects from collections (again, this is fully repeatable given the
            initial seed if there are no race conditions),
          - the runner provides infrastructure to augment stack traces with information about the initial
            seeds used for running the test, so that it can be repeated (or it can be determined that
            the test is not repeatable -- this indicates a problem with the test case itself).

          Thread control:

          - any threads created as part of a test case are assigned the same initial random seed
            (repeatability),
          - tracks and attempts to terminate any Threads that are created and not terminated inside
            a test case (not cleaning up causes a test failure),
          - tracks and attempts to terminate test cases that run for too long (default timeout: 60 seconds,
            adjustable using global property or annotations),

          Improved validation and lifecycle support:

          - RandomizedRunner uses relaxed contracts of hook methods' accessibility (hook methods _can_ be
            private). This helps in avoiding problems with method shadowing (static hooks) or overrides
            that require tedious super.() chaining). Private hooks are always executed and don't affect
            subclasses in any way, period.
          - @Listeners annotation on a test class allows it to hook into the execution progress and listen
            to events.
          - @Validators annotation allows a test class to provide custom validation strategies
            (project-specific). For example a base class can request specific test case naming strategy
            (or reject JUnit3-like methods, for instance).
          - RandomizedRunner does not "chain" or "suppress" exceptions happening during execution of
            a test case (including hooks). All exceptions are reported as soon as they happened and multiple
            failure reports can occur. Most environments we know of then display these failures sequentially
            allowing a clearer understanding of what actually happened first.
          Dawid Weiss made changes -
          Description I love the idea of randomized testing. Everyone (we at CarrotSearch, Lucene and Solr folks) have their glue to make it possible. The question is if there's something to pull out that others could share without having the need to import Lucene-specific classes.
          I love the idea of randomized testing. Everyone (we at CarrotSearch, Lucene and Solr folks) have their glue to make it possible. The question is if there's something to pull out that others could share without having the need to import Lucene-specific classes.

          The work on this issue is on my github account (lots of experiments):
          https://github.com/dweiss/randomizedtesting

          Or directly: git clone git://github.com/dweiss/randomizedtesting.git
          Dawid Weiss made changes -
          Dawid Weiss made changes -
          Field Original Value New Value
          Link This issue relates to LUCENE-3489 [ LUCENE-3489 ]
          Dawid Weiss created issue -

            People

            • Assignee:
              Dawid Weiss
              Reporter:
              Dawid Weiss
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development