[ARTEMIS-1036] Streaming huge messages between cluster nodes causes java.lang.OutOfMemoryError: Direct buffer memory

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.4, 2.0.0
• Component/s: Broker
• Assignee: Francesco Nigro
• Reporter: Francesco Nigro
• Created: 13/Mar/17 17:11
• Updated: 31/Aug/17 19:11
• Resolved: 12/Apr/17 09:21

Description

Scenario

• 2 servers are started in cluster.
• Send one 1GB message to node1. Message is sent as described in subchapter Streaming over JMS in Artemis docs , it is sent like stream from file :

  BytesMessage message = session.createBytesMessage();
  message.setObjectProperty("JMS_AMQ_InputStream", bufferedInput);
  producer.send(message);
  

• Connect consumer on node2 and wait for message to redistribute to node2.
• Receive 1GB stream message from node2 as described in subchapter Streaming over JMS in Artemis docs , it is saved to file.

   BytesMessage messageReceived = (BytesMessage) consumer.receive(timeout);
  // This will block until the entire content is saved on disk
  messageReceived.setObjectProperty("JMS_AMQ_SaveStream", bufferedOutput);
  

What is wrong
During message redistribution from node1 to node2 following error appears. It is thrown on node1.

OOM on node1 during message redistribution

17:52:31,171 WARN  [org.apache.activemq.artemis.utils.OrderedExecutorFactory] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@58d509f4-431458715)) failed to allocate 16777216 byte(s) of direct memory (used: 721420295, max: 734527488): io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 721420295, max: 734527488)
	at ...

Notes

• This scenario only with one server (no cluster, just send/receive on one node) passes OK.
• Message size is larger than server JVM memory. Artemis docs says following : "The only realistic limit to the size of a message that can be sent or consumed is the amount of disk space you have available".

Issue Links

• is blocked by: ARTEMIS-1098 Improve flow control while streaming large messages
• is blocked by: ARTEMIS-1025 Improve flow control on NettyConnection
• links to: GitHub Pull Request #1123
• links to: GitHub Pull Request #1154