[SOLR-8849] ChaosMonkey should cuase chaos in a more reproducible manner

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0
• Assignee: Chris M. Hostetter
• Reporter: Chris M. Hostetter
• Created: 14/Mar/16 23:29
• Updated: 09/May/16 18:54
• Resolved: 15/Mar/16 19:48

Description

Looking into the ChaosMonkey code a bit, and it seems like this class – particularly the way monkeyThread is defined – uses randomness in a way that makes it extremely unlikely that it will ever create reproducible failures.

Obviously in any test where there are multiple concurrent threads, timing issues might prevent test reproducibility – but in this case, even the sequence of "chaos" actions the monkeyThread takes won't be reproducible if anyother concurrent test thread accesses LuceneTestCase.random() ...

      public void run() {
        while (!stop) {
          try {
    
            Random random = LuceneTestCase.random();
            // ... lots of stuff using random, or calling methods that use LuceneTestCase.random() directly

It seems like it would be a lot better if ChaosMonkey's constructor created it's own private Random chaosRand using LuceneTestCase.random() as a seed, and then used chaosRand to make all random choices in it's methods.

That way at least the sequence of chaotic operations made by ChaosMonkey would be consistent for a given test seed, even if the exact timing/interleaving of those operations relative to other operations by other threads couldn't be garunteed.