[OAK-10010] Test org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest.addRemoveSet is Flaky

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 1.46.0
• Component/s: documentmk
• External issue URL: https://github.com/apache/jackrabbit-oak/pull/763
• Assignee: Marcel Reutegger
• Reporter: Yichen
• Created: 23/Nov/22 17:31
• Updated: 21/Dec/22 03:39
• Resolved: 24/Nov/22 07:36

Description

org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest.addRemoveSet 

This is a flaky test, it can pass mvn test while but when run using the tool NonDex, it fails. NonDex is a tool that will introduce non-determinism in certain java collections.The test shows below:

[INFO] Running org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.615 s <<< FAILURE! - in org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest
[ERROR] addRemoveSet(org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest)  Time elapsed: 1.601 s  <<< ERROR!
java.lang.Exception: 
log: 0: [1] + "t8": { "x": 3}
0: [1]  -> r184a581b34a-0-1,r184a581b37b-0-2
0: [1] get t8        at org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest.addRemoveSet(RandomizedClusterTest.java:240)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.junit.runners.Suite.runChild(Suite.java:128)
        at org.junit.runners.Suite.runChild(Suite.java:27)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
        at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: org.junit.ComparisonFailure: expected:<{"[:childNodeCount":0,"x":3]}> but was:<{"[x":3,":childNodeCount":0]}>
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest.get(RandomizedClusterTest.java:330)
        at org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest.get(RandomizedClusterTest.java:303)
        at org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest.addRemoveSet(RandomizedClusterTest.java:220)
        ... 40 more[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   RandomizedClusterTest.addRemoveSet:240  log: 0: [1] + "t8": { "x": 3}
0: [1]  ...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0 

Steps to reproduce the failure:

Run the following command in oak:

First, build the module:

mvn install -pl oak-store-document -am -DskipTests

Then run the test using NonDex 

mvn -pl oak-store-document edu.illinois:nondex-maven-plugin:2.1:nondex -Dtest=org.apache.jackrabbit.oak.plugins.document.RandomizedClusterTest#addRemoveSet

The result will be saved under the module folder in .nondex