Major Description
It is possible to loose an ack after failover if an outstanding consumer transaction (and ack) is in progress during failover. If due to ordering or timing, an unexpected message is replayed to the consumer on recovery it will delivered (and correctly not suppressed as a duplicate). This will be acked with the outstanding messages but the ack will result in an exception
javax.jms.JMSException: Unmatched acknowledege: MessageAck...
as the original message will not have been re-dispatched. Essentially the ack is lost at this stage.
The message will stay dispatched/inflight til the consumer closes, at which point it can again be re-dispatched to another consumer. A broker restart will also see it re-dispatched. In the mean time, it can look orphaned for some time or will be visible in the jdbc store. It will also be visible via the inflight count on that consumer.
Resolution:
On a transport disconnect, a consumer should discard acked state along with delivered messages as the messages that are redelivered are not guaranteed to be the same. This was not being done for a transacted session. Replayed messages are more likely to be the same if the order of connection recovery is preserved, but this will not be sufficient. (the test case shows the problem because recovery order is dependent on hashmap order which is random when dealing with connection ids)
This needs to be done for both consumers that use receive() or message listeners (that are handled through dispatch)