Blocker Description
The recent fixes for rollback QPID-2116 and QPID-1871 highlighted a race condition in the broker.
When messages are rejected asynchronous delivery is started to see if any current consumer wants the message. The 0-8 java client rejects messasges in the following order: Prefetch then Delivered. As a result the RollbackOrderTest#testOnMessage that receives one message then rolls back can receive the second message ahead of the first message.
This was possible because we retrieved the node (getLastSeenNode) then later checked if we were suspended. So with a suitable sleep in the client between rollback and the restarting of the Channel Flow state (=true) the problem can be seen to vanish. What was occuring was the async process retreived the last seen (message 2) the last Reject is processed releasing Message 1. The Async process is still attempting delivery of Message 2 when the Flow=true arrives and so it can deliver Messasge 2.
The fix is to ensure that a suspended client cannot enter a path where it could deliver a message.
This is safe as all Rejects will be full processed before the Flow=true is sent so any process attempting to getLastSeenNode() must do so AFTER a sub.isSuspended() check.