[JAMES-3290] Cassandra exception is not handled in JamesSpooler

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: rabbitmq, SpoolManager & Processors
• Assignee: Unassigned
• Reporter: Gautier DI FOLCO
• Created: 30/Jun/20 15:40
• Updated: 03/Jul/20 10:49
• Resolved: 03/Jul/20 10:49

Description

In the distributed flavor, we rely on MailLoader which itself rely on Cassandra.

Since this code mix-up reactive and non-reactive code, Cassandra errors are non handled which break in the RabbitMQ part:

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra-0 (com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra-0] Timed out waiting for server response), cassandra-1 (com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra-1] Timed out waiting for server response), cassandra-2 (com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra-2] Timed out waiting for server response))
	at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:284)
	at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:376)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:558)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:540)
	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:981)
	at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1582)
	at com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663)
	at com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738)
	at com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466)
	at com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
		at reactor.core.publisher.Mono.block(Mono.java:1666)
		at org.apache.james.queue.rabbitmq.MailLoader.load(MailLoader.java:46)
		at com.github.fge.lambdas.functions.FunctionChainer.lambda$sneakyThrow$49(FunctionChainer.java:74)
		at org.apache.james.queue.rabbitmq.Dequeuer.loadMail(Dequeuer.java:146)
		at org.apache.james.queue.rabbitmq.Dequeuer.loadItem(Dequeuer.java:124)
		at org.apache.james.queue.rabbitmq.Dequeuer.lambda$deQueue$1(Dequeuer.java:105)
		at reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.onNext(FluxMergeSequential.java:201)
		at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
		at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:793)
		at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:718)
		at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153)
		at reactor.rabbitmq.Receiver.lambda$null$9(Receiver.java:190)
		at com.rabbitmq.client.impl.recovery.AutorecoveringChannel$2.handleDelivery(AutorecoveringChannel.java:577)
		at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
		at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:104)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		... 1 common frames omitted
t thread 	pool-17-thread-5

Worse: it actually break the spooling threads, meaning that mails are not dequeued.