[ACCUMULO-3141] Many RW failures due to balance check

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.3, 1.7.1, 1.8.0

Environment

1.5.2 RC1, Hadoop 2.3.0, Zookeeper 3.4.5, CentOS 6, 20 node EC2

• Assignee: Keith Turner
• Reporter: Keith Turner
• Created: 17/Sep/14 19:48
• Updated: 25/Jun/15 19:43
• Resolved: 23/Jun/15 22:39

Description

While running RW test against 1.5.2 RC1, 10 of 17 walkers failed with a message like the following.

16 19:35:48,820 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node Concurrent.xml
        at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
        at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:63)
        at org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:122)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.accumulo.start.Main$1.run(Main.java:107)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.Exception: Error running node ct.CheckBalance
        at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
        at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
        ... 8 more
Caused by: java.lang.Exception: servers are unbalanced! location 2487f8db354002f count 345 too far from average 151.86666666666667
        at org.apache.accumulo.test.randomwalk.concurrent.CheckBalance.visit(CheckBalance.java:86)
        at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)

Issue Links

• is duplicated by: ACCUMULO-2673 Random walk balance check is still failing too frequently
• is related to: ACCUMULO-2488 Concurrent randomwalk balance check needs refinement