Major Description
When sending a message to a topic using a transacted session, memory usage is not decremented correctly when session.rollback is called. It is decremented on commit but not on rollback. This has quite bad consequences since after some time depending on the system memory usage settings and amount of rolled back messages, the broker starts to reject messages saying that Usage Manager memory limit is reached. The only solution is to restart the broker.
I created a small Maven project (see the attachment) with a unit test starting an embedded broker and two test methods: one sending messages with session.commit and another with session.rollback(). The last one fails to assert the memory usage. In the output one can see quite a lot of error logs written by ActiveMQ. The problem is reproducible with 5.8.0 and 5.9.0.
The problem seems to be in org.apache.activemq.broker.region.Topic#doMessageSend method where a transaction synchronization is registered. In the transaction synchronization only afterCommit is supplied, but no afterRollback. So there seems to be nobody calling message.decrementReferenceCount().