[ACCUMULO-2935] Tests consistently hanging after Thrift 0.9.1

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.1, 1.7.0
• Component/s: test
• Assignee: Christopher Tubbs
• Reporter: Josh Elser
• Created: 22/Jun/14 19:21
• Updated: 26/Jun/14 21:54
• Resolved: 25/Jun/14 20:58

Description

Seen MetaSplitIT indefinitely block on my jenkins instance for the past few builds.

TServer Log:

2014-06-22 19:17:13,370 [tablet.Tablet] WARN : Cannot split root tablet
2014-06-22 19:17:13,370 [util.CustomNonBlockingServer$CustomFrameBuffer] ERROR: Unexpected throwable while invoking!
java.lang.RuntimeException: Cannot split root tablet
	at org.apache.accumulo.tserver.tablet.Tablet.split(Tablet.java:2110)
	at org.apache.accumulo.tserver.TabletServer.splitTablet(TabletServer.java:1878)
	at org.apache.accumulo.tserver.TabletServer.access$1400(TabletServer.java:239)
	at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.splitTablet(TabletServer.java:1315)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
	at com.sun.proxy.$Proxy12.splitTablet(Unknown Source)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$splitTablet.getResult(TabletClientService.java:2533)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$splitTablet.getResult(TabletClientService.java:2517)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:169)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
	at org.apache.accumulo.server.util.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:745)

Relevant failsafe runner stack

Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode):
"addSplits 1" daemon prio=10 tid=0x00007f6210154800 nid=0x7c83 waiting on condition [0x00007f61fbdfc000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.accumulo.core.util.UtilWaitThread.sleep(UtilWaitThread.java:26)
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.addSplits(TableOperationsImpl.java:472)
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.access$300(TableOperationsImpl.java:122)
	at org.apache.accumulo.core.client.impl.TableOperationsImpl$SplitTask.run(TableOperationsImpl.java:387)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.lang.Thread.run(Thread.java:745)
"Thread-17" prio=10 tid=0x00007f625c686800 nid=0x7c7e waiting on condition [0x00007f62502b5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c0309398> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282)
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.addSplits(TableOperationsImpl.java:438)
	at org.apache.accumulo.test.MetaSplitIT.testRootTableSplit(MetaSplitIT.java:39)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Issue Links

• is broken by: THRIFT-1805 Thrift should not swallow ALL exceptions
• is broken by: ACCUMULO-1691 Update thrift to 0.9.1 / clean up TServerUtils
• Is contained by: ACCUMULO-2950 Fix handling of server-side exceptions
• is duplicated by: ACCUMULO-2951 ConditionalWriterIT failing on 1.6.1-SNAPSHOT and master
• requires: ACCUMULO-2950 Fix handling of server-side exceptions