[ACCUMULO-3186] ConditionalWriterIT.testThreads failed due to NPE in tserver

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: test
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 30/Sep/14 16:00
• Updated: 20/Oct/14 15:59
• Resolved: 20/Oct/14 15:58

Description

Saw a failure in ConditionalWriterIT last night

org.apache.accumulo.core.client.AccumuloException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server juno:38577
	at org.apache.accumulo.core.client.ConditionalWriter$Result.getStatus(ConditionalWriter.java:61)
	at org.apache.accumulo.test.ConditionalWriterIT.testThreads(ConditionalWriterIT.java:968)
Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server juno:38577
	at org.apache.accumulo.core.client.impl.ConditionalWriterImpl.sendToServer(ConditionalWriterImpl.java:615)
	at org.apache.accumulo.core.client.impl.ConditionalWriterImpl.access$700(ConditionalWriterImpl.java:92)
	at org.apache.accumulo.core.client.impl.ConditionalWriterImpl$SendTask.run(ConditionalWriterImpl.java:452)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.TApplicationException: Internal error processing conditionalUpdate
	at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_conditionalUpdate(TabletClientService.java:526)
	at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.conditionalUpdate(TabletClientService.java:510)
	at org.apache.accumulo.core.client.impl.ConditionalWriterImpl.sendToServer(ConditionalWriterImpl.java:579)
	at org.apache.accumulo.core.client.impl.ConditionalWriterImpl.access$700(ConditionalWriterImpl.java:92)
	at org.apache.accumulo.core.client.impl.ConditionalWriterImpl$SendTask.run(ConditionalWriterImpl.java:452)
	at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Corresponding error on the server appears to be:

2014-09-30 03:30:13,922 [tserver.TabletServer$ThriftClientHandler] ERROR:
java.lang.NullPointerException
        at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.writeConditionalMutations(TabletServer.java:1101)
        at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.conditionalUpdate(TabletServer.java:1203)
        at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.conditionalUpdate(TabletServer.java:1257)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.accumulo.trace.instrument.thrift.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
        at org.apache.accumulo.server.util.RpcWrapper$1.invoke(RpcWrapper.java:44)
        at com.sun.proxy.$Proxy12.conditionalUpdate(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$conditionalUpdate.getResult(TabletClientService.java:2458)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$conditionalUpdate.getResult(TabletClientService.java:2442)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:169)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
        at org.apache.accumulo.server.util.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:745)
2014-09-30 03:30:13,925 [thrift.ProcessFunction] ERROR: Internal error processing conditionalUpdate
org.apache.thrift.TException
        at org.apache.accumulo.server.util.RpcWrapper$1.invoke(RpcWrapper.java:48)
        at com.sun.proxy.$Proxy12.conditionalUpdate(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$conditionalUpdate.getResult(TabletClientService.java:2458)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$conditionalUpdate.getResult(TabletClientService.java:2442)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:169)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
        at org.apache.accumulo.server.util.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:745)

Saw this against master, but marked as fixFor 1.6.2 as well because the two branches haven't really diverged (to my knowledge). Needs some triage if nothing else.

Issue Links

• duplicates: ACCUMULO-3222 Durability obtained before tablet null check
• is broken by: ACCUMULO-1957 specify durability during writes