Major Description
There is a rare issue with memoryUsage when using concurrentStoreAndDispatch that causes memory usage to get out of sync and go negative or stay too high. It is hard to reproduce and only occurs on brokers under high load in my testing.
I have finally tracked down the issue to the InnerFutureTask inside of KahaDB that gets used to fire the PendingMarshalUsageTracker on completion. This task is not thread safe and causes the following 2 problems:
1) Sometimes the listener is never fired and the usage isn't decremented. This case seems to be caused by the listener not being visible to the thread when run. (volatile or atomic reference need to be used)
2) Sometimes the listener can run twice. This seems to be because of a lack of synchronization between the if(isDone()) check and the done() method call. This can also be fixed by using an atomic reference.
I have a patch that solves both of these problems and I have not seen the issue reoccur in my testing.