[AMQ-6596] Out Of Memory error reported on ActiveMQ client during openwire unmarshalling

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 5.13.0, 5.13.5, 5.14.3
• Component/s: Broker, JMS client, Transport

Environment

SUSE Linux Enterprise Server 11 (x86_64), VERSION = 11, PATCHLEVEL = 3. Java Runtime: IBM Corporation 1.7.0

• Assignee: Unassigned
• Reporter: KoPs
• Created: 10/Feb/17 14:48
• Updated: 04/Mar/22 10:41

Description

ActiveMQ crashes during a vulnerability scanning using Qualys vulnerability scanner.

2017-02-10 14:30:18,631 [0.1:55345@61616] WARN  Transport                      - Transport Connection to: tcp://127.0.0.1:55345 failed: java.io.IOException: Unexpected error occurred: java.lang.OutOfMemoryError: Java heap space

This OOM error is caused due to an attempt to initialize a huge byte array of 2131230728 bytes size. The problem still occurs even if we define the maxFrameSize to 100MB.

It seems that the first comparison with the maxFrameSize is successful (method unmarshal) but in the later evaluation of dataIn.readInt() in method looseUnmarshalByteSequence, a huge value is evaluated (2131230728 bytes)

@Override
    public Object unmarshal(DataInput dis) throws IOException {
        DataInput dataIn = dis;
        if (!sizePrefixDisabled) {
            int size = dis.readInt();
            if (size > maxFrameSize) {
                throw IOExceptionSupport.createFrameSizeException(size, maxFrameSize);
            }
            // int size = dis.readInt();
            // byte[] data = new byte[size];
            // dis.readFully(data);
            // bytesIn.restart(data);
            // dataIn = bytesIn;
        }
        return doUnmarshal(dataIn);
    }

    protected ByteSequence looseUnmarshalByteSequence(DataInput dataIn) throws IOException {
        ByteSequence rc = null;
        if (dataIn.readBoolean()) {
            int size = dataIn.readInt();
            byte[] t = new byte[size];
            dataIn.readFully(t);
            rc = new ByteSequence(t, 0, size);
        }
        return rc;
    }

Relevant parts of the thread dump can be found below:

WARNING : OutOfMemoryError possibly caused by 2131230728 bytes requested for object of class 081A5700 from memory space 'Flat' id=080B1898

Thread Name
ActiveMQ Transport: tcp:///10.4.240.10:55345@61616
State
Runnable
Java Stack
at org/apache/activemq/openwire/v12/BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638) 
at org/apache/activemq/openwire/v12/WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132) 
at org/apache/activemq/openwire/OpenWireFormat.doUnmarshal(OpenWireFormat.java:367(Compiled Code)) 
at org/apache/activemq/openwire/OpenWireFormat.unmarshal(OpenWireFormat.java:278(Compiled Code)) 
at org/apache/activemq/transport/tcp/TcpTransport.readCommand(TcpTransport.java:240(Compiled Code)) 
at org/apache/activemq/transport/tcp/TcpTransport.doRun(TcpTransport.java:232(Compiled Code)) 
at org/apache/activemq/transport/tcp/TcpTransport.run(TcpTransport.java:215) 
at java/lang/Thread.run(Thread.java:863)

The definition of the transportConnector without the definition of the maxFrameSize is the following :

        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>

The definition of the transportConnector after the definition of the maxFrameSize :

 <transportConnectors>
      <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?wireFormat.maxFrameSize=104857600"/>
    </transportConnectors>

We have reproduced this with versions 5.13.0, 5.13.5 and 5.14.3 but this problem is probably related to other versions too.