[ARTEMIS-2678] Incomplete records for pages under high load

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: Broker

Environment

Linux

• Assignee: Unassigned
• Reporter: Ansgar J. Sachs
• Created: 23/Mar/20 16:58
• Updated: 11/Oct/21 19:05
• Resolved: 11/Oct/21 19:04

Description

As developer, I expect paging to be resource saving and resilient to high load

Current Situation

During a performance test with a throughput of ~25.000 messages per second that run mulitple hours, some consumers were too slow to consume and messages piled up on the broker. For this reason, the broker started to page the messages of growing queues. When we reduced the load from the broker, some queues stopped consuming due to the following logs:

AMQ222033: Page file 000000007.page had incomplete records at position 39,795,399 at record number 13,952

target message no.16146 not found from start offset 46032883 and start message number 16146: java.lang.RuntimeException: target message no.16146 not found from start offset 46032883 and start message number 16146

It wasnt possible to recover from this state but deleting the paging directory.

Expected Situation

I would expect that the paging mechanism is resilient to any errors.

Scenario Setup

Master configuration:

<ha-policy>
  <shared-store>
    <master>
      <failover-on-shutdown>true</failover-on-shutdown>
    </master>
  </shared-store>
</ha-policy>
<!-- ... -->
 <address-setting match="#">
        <max-size-bytes>256Mb</max-size-bytes>
        <page-size-bytes>64Mb</page-size-bytes>
        <message-counter-history-day-limit>10</message-counter-history-day-limit>
        <address-full-policy>PAGE</address-full-policy>
</address-setting>

An extract of the monitoring of the Performance Test is attached to this issue.

Workaround

Right now we disabled paging at all and only use our Heap. However, the heap is exhausted at 5 million messages which is in our use case better than loosing any of them.