[JAMES-3726] Allow relaxing CassandraMailRepository LWT usage

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.8.0
• Component/s: cassandra, MailStore & MailRepository
• Assignee: Unassigned
• Reporter: Benoit Tellier
• Created: 15/Mar/22 05:02
• Updated: 30/Mar/22 03:56
• Resolved: 30/Mar/22 03:56

Description

At a customer place I discovered CassandraMailRepository extensive relies on lightweight transactions, which turns out not playing well when doing massive error recovery operations on mail repositories:

2022-03-10T11:09:40,137 [ERROR] o.a.j.m.i.JamesMailSpooler - Failed Mail1645005443013-a8c3d070-afd7-4821-990d-7c309ff38970 processing 1 consecutive times. Mail is requeued with increased failure count.
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during CAS write query at consistency LOCAL_SERIAL (2 replica were required but only 1 acknowledged the write)
	at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87)
	at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65)
	at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297)
	at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268)
	at com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	... 25 common frames omitted
Wrapped by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during CAS write query at consistency LOCAL_SERIAL (2 replica were required but only 1 acknowledged the write)
	at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113)
	at com.datastax.driver.core.Responses$Error.asException(Responses.java:167)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
		at reactor.core.publisher.Mono.block(Mono.java:1703)
		at org.apache.james.mailrepository.cassandra.CassandraMailRepository.store(CassandraMailRepository.java:74)
		at org.apache.james.transport.mailets.ToRepository.service(ToRepository.java:79)
		at org.apache.james.mailetcontainer.impl.camel.CamelProcessor.process(CamelProcessor.java:75)
		at org.apache.james.mailetcontainer.impl.camel.CamelMailetProcessor$MailetContainerRouteBuilder.handleMailet(CamelMailetProcessor.java:176)
		at org.apache.james.mailetcontainer.impl.camel.CamelMailetProcessor$MailetContainerRouteBuilder.lambda$configure$0(CamelMailetProcessor.java:153)

In my opinion these LWT are far from relevant and could be replaced by an idempotent behaviour. We should expose a configuration option allowing to do just that.

Proposal: cassandra.properties

mailrepository.strong.consistency=false

Issue Links

• links to: GitHub Pull Request #921