[QPID-7753] Sparsely occupied message buffers may lead to java.lang.OutOfMemoryError: Direct buffer memory

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: qpid-java-6.0, qpid-java-6.1
• Fix Version/s: qpid-java-broker-7.0.0
• Component/s: Broker-J
• Assignee: Unassigned
• Reporter: Keith Wall
• Created: 21/Apr/17 14:13
• Updated: 02/Oct/19 19:22
• Resolved: 30/May/17 09:47

Description

Some Broker usage patterns can lead to the Broker failing with a "java.lang.OutOfMemoryError: Direct buffer memory" error.

For the condition to manifest a producing application needs to use a single connection for the production of messages. Some messages need to be consumed quickly whilst others remain on the Broker. This pattern might result from:

1. the consuming application using message selectors to selectively consume some messages whilst leaving others on the Broker.
2. the use of 'out of order' queue types (priority/sorted etc) where lower priority items are left of the Broker.
3. the producing application routing messages to multiple queues and the consuming application draining some queues but not others.

The problem arises owing to the buffering strategy under the IO layer.

NonBlockingConnection allocates a netInputBuffer from pooled direct memory of size 256K. This buffer is used for all network reads until the space remaining in the buffer is less than the amount required to complete the AMQP frame that is currently being read, in which case a new netInputBuffer is allocated. The protocol layers identify the message payload/message headers as part of AMQP frame parsing and create a byte-buffer views onto the original input byte buffer. This byte buffer is retained by the store until the message is consumed. In the usage pattern described above, a single long lived message amongst a stream of shorted lived messages causes an entire 256K buffer chunk to be retained. Qpid cannot dispose or reuse the buffer until it is entirely unoccupied.

The flow to disk feature is designed to prevent excessive direct memory use by flushing messages to disk if thresholds are breached. Flow to disk does not help the scenario described above because it considers the total payloads of live messages. Its algorithm does not consider direct memory occupancy.

Issue Links

• is related to: QPID-7754 Producing application using a connection per message pattern for small long lived messages may crash the Broker: java.lang.OutOfMemoryError: Direct buffer memory