[CASSANDRA-10756] Timeout failures in NativeTransportService.testConcurrentDestroys unit test

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.6
• Component/s: Legacy/Testing, Local/Startup and Shutdown
• Severity: Normal
• Assignee: Alex Petrov
• Reporter: Joel Knighton

Authors

Alex Petrov

Reviewers

Joel Knighton

• Tester: Michael Shuler
• Created: 23/Nov/15 04:14
• Updated: 16/Apr/19 09:30
• Resolved: 25/Apr/16 08:01

Description

History of test on trunk here.

I've seen these failures across 3.0/trunk for a while. I ran the test looping locally for a while and the timeout is fairly easy to reproduce. The timeout appears to be an indefinite hang and not a timing issue.

When the timeout occurs, the following stack trace is at the end of the logs for the unit test.

ERROR [ForkJoinPool.commonPool-worker-1] 2015-11-22 21:30:53,635 Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:745) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:322) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:728) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.util.concurrent.DefaultPromise.execute(DefaultPromise.java:671) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyLateListener(DefaultPromise.java:641) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:138) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.group.DefaultChannelGroupFuture.<init>(DefaultChannelGroupFuture.java:116) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.group.DefaultChannelGroup.close(DefaultChannelGroup.java:275) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at io.netty.channel.group.DefaultChannelGroup.close(DefaultChannelGroup.java:167) [netty-all-4.0.23.Final.jar:4.0.23.Final]
	at org.apache.cassandra.transport.Server$ConnectionTracker.closeAll(Server.java:277) [main/:na]
	at org.apache.cassandra.transport.Server.close(Server.java:180) [main/:na]
	at org.apache.cassandra.transport.Server.stop(Server.java:116) [main/:na]
	at java.util.Collections$SingletonSet.forEach(Collections.java:4767) ~[na:1.8.0_60]
	at org.apache.cassandra.service.NativeTransportService.stop(NativeTransportService.java:136) ~[main/:na]
	at org.apache.cassandra.service.NativeTransportService.destroy(NativeTransportService.java:144) ~[main/:na]
	at org.apache.cassandra.service.NativeTransportServiceTest.lambda$withService$102(NativeTransportServiceTest.java:201) ~[classes/:na]
	at java.util.stream.IntPipeline$3$1.accept(IntPipeline.java:233) ~[na:1.8.0_60]
	at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110) ~[na:1.8.0_60]
	at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693) ~[na:1.8.0_60]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_60]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_60]
	at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) ~[na:1.8.0_60]
	at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) ~[na:1.8.0_60]
	at java.util.stream.AbstractTask.compute(AbstractTask.java:316) ~[na:1.8.0_60]
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[na:1.8.0_60]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[na:1.8.0_60]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[na:1.8.0_60]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[na:1.8.0_60]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[na:1.8.0_60]

Issue Links

• links to: patch