Lucene - Core
  1. Lucene - Core
  2. LUCENE-6437

Improve test reproducibility of ConcurrentMergeScheduler

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      If things go wrong based on merges with concurrentmergescheduler in tests, its always more difficult to reproduce.

      But we make it worse, CMS can base its behavior on:

      • number of cpus.
      • whether drives are SSDs.

      We should just allow a (undocumented, for testing) sysprop backdoor to override both of these. This way, it can be set to values completely based on the random seed in LuceneTestCase init, and its not a reproducibility trap.

      Its true we try to set explicit values for LuceneTestCase.newIndexWriterConfig() and so forth. But not all tests use randomized IW configs.

      1. LUCENE-6437.patch
        4 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch. It adds the properties to CMS, and then sets random values for CPU core count and spins derived from the master seed ...

        Show
        Michael McCandless added a comment - Patch. It adds the properties to CMS, and then sets random values for CPU core count and spins derived from the master seed ...
        Hide
        Robert Muir added a comment -

        +1, this is great.

        I think we just need tweaks to test.policy to add write access to the two new permissions. +1 to commit with that.

        Show
        Robert Muir added a comment - +1, this is great. I think we just need tweaks to test.policy to add write access to the two new permissions. +1 to commit with that.
        Hide
        ASF subversion and git services added a comment -

        Commit 1674535 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1674535 ]

        LUCENE-6437: randomize cpu core count and spins for ConcurrentMergeScheduler's dynamic defaults

        Show
        ASF subversion and git services added a comment - Commit 1674535 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1674535 ] LUCENE-6437 : randomize cpu core count and spins for ConcurrentMergeScheduler's dynamic defaults
        Hide
        ASF subversion and git services added a comment -

        Commit 1674537 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1674537 ]

        LUCENE-6437: randomize cpu core count and spins for ConcurrentMergeScheduler's dynamic defaults

        Show
        ASF subversion and git services added a comment - Commit 1674537 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1674537 ] LUCENE-6437 : randomize cpu core count and spins for ConcurrentMergeScheduler's dynamic defaults
        Hide
        Robert Muir added a comment -

        Thanks Mike!

        Show
        Robert Muir added a comment - Thanks Mike!
        Hide
        Simon Willnauer added a comment -

        cool stuff guys!

        Show
        Simon Willnauer added a comment - cool stuff guys!
        Hide
        Anshum Gupta added a comment -

        Bulk close for 5.2.0.

        Show
        Anshum Gupta added a comment - Bulk close for 5.2.0.

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development