Major Description
Hi,
I tried to delete all my scheduled messages and I noticed that the broker stop responding and a huge transaction file was created.
The Scheduled.db file had ~150mb and the transaction was already 140GB when I give up and wiped the broker.
Looking at the code, I noticed that the cause of it is mostly this 2 lines:
And
On the first case, we are updating an List<Int> inside a transaction but in order to do that, we get its value from the BTree and put back (with an added element). When we do that, the whole list is again serialized and written inside the transaction.
Similar thing is happening on the second case, We are decrementing one by a number of references to a journal file but we could do it only one time in the end saving lots of writes.
After the proposed patch, the transaction that was before more than 140GB (and it didn't finish) was reduced to ~500 mb, and the broker could recover in less than 1 minute (i gave up before with ~15 minutes).
On the modified test in the PR we can see that the transaction before the change is ~50mb and after the change is ~500k
Before:
2019-06-03 23:58:19,931 //localhost#1-1 - DEBUG Transaction - Committing transaction 5177: Size 53320 kb
After:
2019-06-03 23:59:13,578 //localhost#1-2 - DEBUG Transaction - Committing transaction 5178: Size 496 kb
PS: Keeping removedJobFileIds in memory will not increase the memory usage bc we were loading it in memory anyway when reading from the index.
Attachments
Issue Links
- links to
