[ACCUMULO-893] splitting and hold time are interacting badly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.2
• Component/s: tserver
• Assignee: Keith Turner
• Reporter: Eric C. Newton
• Created: 06/Dec/12 21:18
• Updated: 20/Feb/14 23:21
• Resolved: 14/Mar/13 20:50

Description

Testing on a medium sized cluster using the continuous ingest test.

I pre-split the table and started ingest. I got tired of waiting for the tablets to split again so I lowered the split threshold to 100M.

The tablet servers started minor compaction in preparation for the split; memory became held.. and then nothing would happen until this error came up:

2012-12-06 12:57:02,025 [thrift.TabletClientService$Processor] ERROR: tserver:r01n16 Internal error processing closeUpdate
org.apache.accumulo.server.tabletserver.HoldTimeoutException: Commits are held
        at org.apache.accumulo.server.tabletserver.TabletServerResourceManager.waitUntilCommitsAreEnabled(TabletServerResourceManager.java:386)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.flush(TabletServer.java:1469)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.closeUpdate(TabletServer.java:1610)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy2.closeUpdate(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.process(TabletClientService.java:2338)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)

The tservers appeared to make no progress on splitting until I stopped the ingesters.

Issue Links

• duplicates: ACCUMULO-1062 large numbers of threads make in-memory updates slow