[CASSANDRA-18380] CEP-15 (Accord) accord.coordinate.tracking.ReadTracker.recordInFlightRead fails with {number} already in flight and Nothing in flight for {number}

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.x
• Component/s: Accord
• Bug Category: Correctness - Unrecoverable Corruption / Loss
• Severity: Normal
• Complexity: Normal
• Discovered By: Fuzz Test
• Platform: All
• Assignee: Unassigned
• Reporter: David Capwell
• Created: 28/Mar/23 20:57
• Updated: 28/Mar/23 20:58

Description

java.lang.IllegalStateException: 1684148777 already in flight
	at accord.coordinate.tracking.ReadTracker.recordInFlightRead(ReadTracker.java:178)
	at accord.coordinate.tracking.ReadTracker.trySendMore(ReadTracker.java:284)
	at accord.coordinate.ReadCoordinator.trySendMore(ReadCoordinator.java:206)
	at accord.coordinate.tracking.AbstractTracker$ShardOutcomes.toRequestStatus(AbstractTracker.java:74)
	at accord.coordinate.tracking.AbstractTracker$ShardOutcomes.access$100(AbstractTracker.java:39)
	at accord.coordinate.tracking.AbstractTracker.recordResponse(AbstractTracker.java:137)
	at accord.coordinate.tracking.AbstractTracker.recordResponse(AbstractTracker.java:123)
	at accord.coordinate.tracking.ReadTracker.recordResponse(ReadTracker.java:235)
	at accord.coordinate.tracking.ReadTracker.recordQuorumReadSuccess(ReadTracker.java:221)
	at accord.coordinate.ReadCoordinator.onSuccess(ReadCoordinator.java:115)
	at accord.coordinate.ReadCoordinator.onSuccess(ReadCoordinator.java:36)
	at accord.messages.CallbackHolder.lambda$success$0(CallbackHolder.java:39)
	at accord.messages.CallbackHolder.lambda$safeCall$3(CallbackHolder.java:62)
	at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:343)
	at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:330)
	at org.apache.cassandra.service.accord.async.AsyncOperation.runInternal(AsyncOperation.java:226)
	at org.apache.cassandra.service.accord.async.AsyncOperation.run(AsyncOperation.java:268)

and

java.lang.IllegalStateException: Nothing in flight for 1684873295
	at accord.coordinate.tracking.ReadTracker.receiveResponseIsSlow(ReadTracker.java:186)
	at accord.coordinate.tracking.ReadTracker.recordResponse(ReadTracker.java:234)
	at accord.coordinate.tracking.ReadTracker.recordReadSuccess(ReadTracker.java:213)
	at accord.coordinate.ReadCoordinator.onSuccess(ReadCoordinator.java:119)
	at accord.coordinate.ReadCoordinator.onSuccess(ReadCoordinator.java:36)
	at accord.messages.CallbackHolder.lambda$success$0(CallbackHolder.java:39)
	at accord.messages.CallbackHolder.lambda$safeCall$3(CallbackHolder.java:62)
	at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:343)
	at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:330)
	at org.apache.cassandra.service.accord.async.AsyncOperation.runInternal(AsyncOperation.java:226)
	at org.apache.cassandra.service.accord.async.AsyncOperation.run(AsyncOperation.java:268)
	at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
	Suppressed: accord.coordinate.Timeout
		at org.apache.cassandra.service.accord.AccordCallback.convertReason(AccordCallback.java:53)
		at org.apache.cassandra.service.accord.AccordCallback.onFailure(AccordCallback.java:63)
		at org.apache.cassandra.net.RequestCallbacks.lambda$onExpired$0(RequestCallbacks.java:173)
		... 7 more
		Suppressed: accord.coordinate.Timeout
			... 10 more

Issue Links

• blocks: CASSANDRA-18196 Initial Merge of Accord Feature Branch to Trunk