[ACCUMULO-2438] deleting all table rows caused tablet load failure

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Fix Version/s: 1.5.2, 1.6.0
• Component/s: master
• Labels: 16_qa_bug

Environment

nightly single-node test

• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 07/Mar/14 15:15
• Updated: 11/Mar/14 16:30
• Resolved: 10/Mar/14 03:51

Description

testTrace in ConditionalWriterIT failed:

java.lang.Exception: test timed out after 60000 milliseconds
	at java.lang.Thread.sleep(Native Method)
	at org.apache.accumulo.core.util.UtilWaitThread.sleep(UtilWaitThread.java:26)
	at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:240)
	at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:84)
	at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:177)
	at org.apache.accumulo.core.trace.TraceDump.printTrace(TraceDump.java:133)
	at org.apache.accumulo.test.ConditionalWriterIT.testTrace(ConditionalWriterIT.java:1231)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:622)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Examining the logs, it seems like the client was having a hard time locating the trace table (table id 9):

2014-03-07 04:33:40,445 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 9 row : 9dd07d1bd8b9bc0d
...
2014-03-07 04:33:40,551 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 9 row : 9dd07d1bd8b9bc0d

Examining the master logs, I see the trace table being created, assigned, and hosted.

This test deletes everything in the trace table before tracing an operation:

    conn.tableOperations().deleteRows("trace", null, null);

The master takes the table offline and re-writes the metadata table information so that it no longer has any data.

The master then attempts to put the tablet online, but the operation fails:

2014-03-07 04:33:40,079 [master.Master] ERROR: host1:47630 reports assignment failed for tablet 9<<

The tablet server gives the real problem:

2014-03-07 04:33:40,074 [tserver.TabletServer] WARN : exception trying to assign tablet 9<< file:////local/disk1/jenkins/workspace/accumulo16/test/target/mini-tests/org.apache.accumulo.test.functional.SimpleMacIT/1394184783304_4048/accumulo/tables/9/default_tablet
java.lang.IllegalArgumentException: Time type unknown : ^@0
        at org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:67)
        at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1296)
        at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1211)
        at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1067)
        at org.apache.accumulo.tserver.Tablet.<init>(Tablet.java:1056)
        at org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2911)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:701)

Tablet time should always be "L" or "M".

Issue Links

• is related to: ACCUMULO-2450 Backport ACCUMULO-2438 to 1.4