[ACCUMULO-3794] Replication randomwalk module failed on verification

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: replication, test
• Labels: 1.7.0_QA
• Assignee: Eric C. Newton
• Reporter: Josh Elser
• Created: 11/May/15 15:39
• Updated: 06/Mar/20 02:11
• Resolved: 06/Mar/20 02:11

Description

11 03:37:45,852 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node Concurrent.xml
        at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
        at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
        at org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.accumulo.start.Main$2.run(Main.java:130)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Error running node ct.Replication
        at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
        at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
        at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        ... 1 more
Caused by: java.lang.RuntimeException: 0 fails to match expected value 1000
        at org.apache.accumulo.test.randomwalk.concurrent.Replication.assertEquals(Replication.java:180)
        at org.apache.accumulo.test.randomwalk.concurrent.Replication.visit(Replication.java:167)
        ... 9 more

Leading up to this, the test was waiting on a WAL to be replicated

11 03:37:15,334 [impl.ReplicationOperationsImpl] DEBUG: Collecting referenced files for replication of table repl_source_026fb5bb_b94c_4a7b_8480_d649d7b383f5
11 03:37:15,334 [impl.ReplicationOperationsImpl] DEBUG: Found id of 51 for name repl_source_026fb5bb_b94c_4a7b_8480_d649d7b383f5
11 03:37:15,345 [concurrent.Replication] DEBUG: updateFileRefs size 1
11 03:37:15,707 [impl.ReplicationOperationsImpl] DEBUG: Collecting referenced files for replication of table repl_source_026fb5bb_b94c_4a7b_8480_d649d7b383f5
11 03:37:15,707 [impl.ReplicationOperationsImpl] DEBUG: Found id of 51 for name repl_source_026fb5bb_b94c_4a7b_8480_d649d7b383f5
11 03:37:15,713 [concurrent.Replication] DEBUG: updateFileRefs size 0

It would seem that we waited on a WAL correctly, but the WAL we waiting on didn't contain the records we were waiting for (or there's a bug elsewhere).