Implementing this part alone is really trivial and was very quickly done, but the problem is, that this is not the only limitation why the same consumer instance must be used. So this alone would be of no value.
Currently the ActiveMQMessageConcumser is scheduling a task to perform the relivery. If there are multiple consumers, another consumer will simply grab the same message arriving again from the broker and do a normal delivery (so without any delay and completely asynchrounous to the scheduled thread still waiting). At some point I have not yet clearly identified the broker must get an ACK or something like this. Maybe as a result of a consumer or session close.
So far I have just experimented with the code to get a better understanding on how this was ever intended to work correctly (especially if using a transaction manager to mange the transactions explicitely and not letting AMQ automatically perform the commit or rollback based on exceptions with a transacted session).
According to the design document at http://activemq.apache.org/message-redelivery-and-dlq-handling.html
" [...] On rollback, since nothing has been acked yet, and all messages are still available in an internal consumer queue, the messages are re-dispatched from the internal consumer queue. This reduces redelivery dispatch overhead, at the cost of the broker not being aware that redeliveries are occurring. [...]"
since ActiveMQ 4 the broker shall not be aware of redeliveries at all and only the consumer should perform the redelivery from a local queue to possibly relieve the broker from some work improving scalability. Unfortunately if I got it right the end user cannot choose between either client or broker controlled redelivery (or 4.x versus 3.x behavior).
In order to make this work I would only see the two theoretical options (not sure whether any of those are correct and feasible):
a) The broker will not be informed about the redeliveries as a result of a rollback at all (only indirectly, if optionally persisting modified messages (redeliverey counter). As long as a consumer is not acknowledging the broker will not deliver the same message to another consumer. But I think incase of a transacted session, such an acknowledgement must be sent, or?
b) The broker will be informed about rollbacks in order to trigger a resend of the same message. Any consumer receiving a message will first check the current redelivery count and postpone the actual delivery of this message for a calculated duration if required. If the redelivery count is 0, it will of course be processed immediately.
As I could not quickly detect the place where the broker gets signaled that it shall resend the message for which a rollback was done, I simply "hacked a bit of code" for the purpose of testing and understanding, which performs the conditional wait based on the modified RedeliveryPolicy calculating the delay based on the counter right inside the ActiveMQSessionExecutor#execute prior dispatching the message. So any message arriving at this point will be hold of from dispatching to the local queue until the calculated delay according to the redelivery attempt has passed. Of course in parallel I commented the scheduled redelivery task for asynchronous execution. So this is something a long the line of option b) mentioned above.
And this is the first time I could see a working redelivery with multiple consumer threads using the PooledConnectionFactory, an external TransactionManager, and neither cache level, sessionAcknowledgeMode nor sessionTransacted configured on Spring's DefaultMessageListenerContainer:
DEBUG 2011-11-17 19:14:04,687 [primary-1] TransactionContext 'Begin:TX:ID:ldev3001-32202-1321553518048-0:1:1'
DEBUG 2011-11-17 19:14:05,296 [primary-3] TransactionContext 'Rollback: TX:ID:ldev3001-32202-1321553518048-0:1:1 syncCount: 2'
DEBUG 2011-11-17 19:14:06,322 [primary-1] TransactionContext 'Begin:TX:ID:ldev3001-32202-1321553518048-0:3:1'
DEBUG 2011-11-17 19:14:06,357 [primary-4] TransactionContext 'Rollback: TX:ID:ldev3001-32202-1321553518048-0:3:1 syncCount: 2'
DEBUG 2011-11-17 19:14:09,375 [primary-1] TransactionContext 'Begin:TX:ID:ldev3001-32202-1321553518048-0:4:1'
DEBUG 2011-11-17 19:14:09,410 [primary-5] TransactionContext 'Rollback: TX:ID:ldev3001-32202-1321553518048-0:4:1 syncCount: 2'
DEBUG 2011-11-17 19:14:18,423 [primary-1] TransactionContext 'Begin:TX:ID:ldev3001-32202-1321553518048-0:5:1'
DEBUG 2011-11-17 19:14:18,442 [primary-6] TransactionContext 'Rollback: TX:ID:ldev3001-32202-1321553518048-0:5:1 syncCount: 2'
DEBUG 2011-11-17 19:14:45,456 [primary-1] TransactionContext 'Begin:TX:ID:ldev3001-32202-1321553518048-0:6:1'
DEBUG 2011-11-17 19:14:45,479 [primary-7] TransactionContext 'Rollback: TX:ID:ldev3001-32202-1321553518048-0:6:1 syncCount: 2'
DEBUG 2011-11-17 19:16:06,491 [primary-1] TransactionContext 'Begin:TX:ID:ldev3001-32202-1321553518048-0:7:1'
DEBUG 2011-11-17 19:16:06,533 [primary-8] TransactionContext 'Rollback: TX:ID:ldev3001-32202-1321553518048-0:7:1 syncCount: 2'
It is probably needless to say, that this approach is neither clean nor likely to work with CACHE_CONSUMER as I don't think the broker will be "triggered" in this case to do a resend.
I did not test this though, as I could not get cache level CACHE_CONSUMER in conjunction with an external transaction manager (which I can't avoid for my use case) to work, as the session needs to be transacted and is only transacted if either having XA Transactions (but I use a LocalTransaction) OR the acknowledgeMode is SESSION_TRANSACTED (which does not seem to be achievable in conjunction with CACHE_CONSUMER, unless setting sessionTransacted to true (which again does not work in conjunction with an external transaction manager).
It would be great if someone with more knowledge about JMS could describe the correct behavior if one wants to achieve a correctly working transaction handling using an external transaction manager with a local transaction where redelivery works as descired and best possilbe performance is reached.
Once I know the desired flow, I can check with the code what it currently does to then find out what to potentially adjust. At the moment I'm really struggling through a quite feature rich and rather complex implementation.
Although we would be happy to contribute improvements, it is very hard without at least some pointers from other more experienced devs from the AMQ team.
Many thanks for anyone helping on this!
If we do not find a way to move forward, will very likely have to try out the same scenario with another JMS broker like JBoss HornetQ and have a look at their implementation. Although I think at the moment they only support a constant redelivery-delay, which is not very suiteable either and would need to be improved for most real world use cases. But such a change is very small compared to completely modifying the general handling of redeliveries.