[ACCUMULO-3304] Add introspection of long running assignments

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.2, 1.7.0
• Component/s: tserver
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 06/Nov/14 00:49
• Updated: 10/Feb/15 23:16
• Resolved: 08/Nov/14 21:52

Description

At least once, I've seen a tablet assignment hang for some inexplicable reason. We could track the active compaction and report on assignments that are taking an excessive amount of time.

Assignment for 54;7;6 has been running for at least 13445ms.
	java.util.zip.Deflater.deflateBytes(Native Method)
	java.util.zip.Deflater.deflate(Deflater.java:430)
	java.util.zip.Deflater.deflate(Deflater.java:352)
	org.apache.hadoop.io.compress.zlib.BuiltInZlibDeflater.compress(BuiltInZlibDeflater.java:54)
	org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81)
	org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:76)
	org.apache.accumulo.core.file.rfile.bcfile.Compression$FinishOnFlushCompressionStream.write(Compression.java:59)
	java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
	java.io.DataOutputStream.write(DataOutputStream.java:107)
	java.io.DataOutputStream.write(DataOutputStream.java:107)
	org.apache.accumulo.core.data.Value.write(Value.java:163)
	org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:388)
	org.apache.accumulo.tserver.Compactor.compactLocalityGroup(Compactor.java:504)
	org.apache.accumulo.tserver.Compactor.call(Compactor.java:362)
	org.apache.accumulo.tserver.MinorCompactor.call(MinorCompactor.java:96)
	org.apache.accumulo.tserver.Tablet.minorCompact(Tablet.java:2071)
	org.apache.accumulo.tserver.Tablet.access$4400(Tablet.java:174)
	org.apache.accumulo.tserver.Tablet$MinorCompactionTask.run(Tablet.java:2158)
	org.apache.accumulo.tserver.Tablet.minorCompactNow(Tablet.java:2267)
	org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2937)
	org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:55)
	org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
	org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
	java.lang.Thread.run(Thread.java:745)

Issue Links

• breaks: ACCUMULO-3576 Usage of jline.internal.Preconditions breaks mini accumulo
• is related to: ACCUMULO-3276 Shard.xml hung with no client output
• links to: RB