[AMQ-8351] MaxFrameSize is not protecting against allocating big buffer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 5.12.2, 5.13.5, 5.14.5, 5.15.14, 5.16.2, 5.17.5
• Component/s: Transport
• Assignee: Unassigned
• Reporter: Marcin
• Created: 06/Aug/21 11:49
• Updated: 05/Jul/23 12:52
• Resolved: 05/Jul/23 11:28

Description

I found that some incoming data can allocate much bigger buffer (up to max int size) than frame size limit. This can lead to oom. I created junit test to show the problem:

 

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInput;
import java.io.DataInputStream;
import java.io.IOException;

import com.google.common.primitives.Ints;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;

class OpenWireFormatTest {

@Test
void maxFrameSizeTest() {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
try

{ outputStream.write(Ints.toByteArray(999)); outputStream.write(1); outputStream.write(Ints.toByteArray(2000000000)); outputStream.write(Ints.toByteArray(2000000000)); outputStream.write(Ints.toByteArray(2000000000)); outputStream.write(1); outputStream.write(Ints.toByteArray(2000000000)); outputStream.write(Ints.toByteArray(2000000000)); outputStream.write(Ints.toByteArray(2000000000)); outputStream.write(Ints.toByteArray(2000000000)); outputStream.write(Ints.toByteArray(2000000000)); }

catch (IOException e)

{ e.printStackTrace(); }

ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(outputStream.toByteArray());
DataInput dataInput = new DataInputStream(byteArrayInputStream);

OpenWireFormat openWireFormat = new OpenWireFormat();
openWireFormat.setMaxFrameSize(1000);

final IOException ioException = assertThrows(IOException.class, () -> openWireFormat.unmarshal(dataInput));
assertEquals("Frame size of 1907 MB larger than max allowed 100 MB", ioException.getMessage());
}