[ACCUMULO-2466] Bulk randomwalk fails with bad key

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.4.4
• Component/s: master, test
• Labels: import randomwalk test
• Assignee: Unassigned
• Reporter: Bill Havanki
• Created: 13/Mar/14 16:06
• Updated: 18/Jul/14 03:54
• Resolved: 18/Jul/14 03:54

Description

Running bulk randomwalk against 1.4.5-SNAPSHOT, got this in verification:

Caused by: java.lang.Exception: Bad key at r00000 cf:000 [] 1394658887772 false 1
        at org.apache.accumulo.server.test.randomwalk.bulk.Verify.visit(Verify.java:65)

Possible reasons:

• ACCUMULO-2110, not backported to 1.4 or 1.5
• master agitation

I see in the logs three internal errors from imports that failed due to the masters being restarted. The failure timing is around 5 seconds after the masters restart. Example:

12 14:10:17,580 [bulk.BulkMinusOne] ERROR: org.apache.accumulo.core.client.AccumuloException: Intern
al error processing waitForTableOperation
org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForTableOperation
        at org.apache.accumulo.core.client.admin.TableOperationsImpl.doTableOperation(TableOperation
sImpl.java:290)
        at org.apache.accumulo.core.client.admin.TableOperationsImpl.doTableOperation(TableOperation
sImpl.java:258)
        at org.apache.accumulo.core.client.admin.TableOperationsImpl.importDirectory(TableOperations
Impl.java:947)
        at org.apache.accumulo.server.test.randomwalk.bulk.BulkPlusOne.bulkLoadLots(BulkPlusOne.java
:99)
        at org.apache.accumulo.server.test.randomwalk.bulk.BulkMinusOne.runLater(BulkMinusOne.java:2
9)
...
Caused by: org.apache.thrift.TApplicationException: Internal error processing waitForTableOperation

Two BulkMinusOne and one BulkPlusOne failed, which may be why the offending row was at value 1.

The TableOperationsImpl.waitForTableOperation method does not catch TApplicationException, so the imports fail.

I see lots of previous work on this sort of error in ACCUMULO-334 and ACCUMULO-2110. If anyone has troubleshooting tips I'd be happy to hear them.

Issue Links

• relates to: ACCUMULO-334 Bulk random walk test failed
• relates to: ACCUMULO-2110 [RW] Error in Bulk.Verify