[OLINGO-1504] JVM crashes due to OutOfMemory encountered: Java heap space

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: (Java) V4 4.7.0
• Component/s: odata4-client
• Assignee: Unassigned
• Reporter: Devansh Soni
• Created: 23/Dec/20 19:54
• Updated: 12/Jul/23 15:33

Description

Hi 
The issue occurs for non-paginated OData feeds. The feed I had tested had 100,000 rows and 9 columns. The JVM crashes due to insufficient heap size and I can find the stack trace from the hs_err_pidPID log file. 

ID	Value
26	Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
27	j  java.util.Arrays.copyOf([BI)[B+1
28	j  java.io.ByteArrayOutputStream.grow(I)V+36
29	j  java.io.ByteArrayOutputStream.ensureCapacity(I)V+12
30	j  java.io.ByteArrayOutputStream.write([BII)V+38
31	j  org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;[B)J+19
32	j  org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;I)J+5
33	j  org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J+5
34	j  org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I+2
35	j  org.apache.olingo.client.core.communication.response.AbstractODataResponse.getRawResponse()Ljava/io/InputStream;+136
36	j  org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Lorg/apache/olingo/client/api/domain/ClientEntitySetIterator;+23
37	j  org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Ljava/lang/Object;+1
38	j  com.tableausoftware.odata.ODataProtocolImpl.fetchV4(Ljava/net/URI;Z)Lcom/tableausoftware/odata/ODataProtocolImpl$ODataResults;+50
39	j  com.tableausoftware.odata.ODataResultSetV4.nextBlockImpl()Lcom/tableausoftware/data/generated/DataStream$Block;+23
40	j  com.tableausoftware.data.ProtobufResultSet.nextBlock()Lcom/tableausoftware/data/generated/DataStream$Block;+1
41	j  com.tableau.connect.service.QueryTask.readData()V+46
42	j  com.tableau.connect.service.QueryTask.call()Ljava/lang/Void;+9
43	j  com.tableau.connect.service.QueryTask.call()Ljava/lang/Object;+1
44	j  java.util.concurrent.FutureTask.run()V+42
45	j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
46	j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
47	j  java.lang.Thread.run()V+11
48	v  ~StubRoutines::call_stub

 

The issue happens because there are multiple copies of streams being created in the AbstractODataResponse.getRawResponse method, specifically in the org.apache.commons.io.IOUtils.copy method. To me it seems like it fails to expand the internal byte buffer when it reaches capacity. 
However, I do not understand why is the response payload being copied in a ByteArrayOutputStream 

org.apache.commons.io.IOUtils.copy(payload, byteArrayOutputStream);

and then again converted into a ByteArrayInputStream. This copying of streams causes creation of multiple byte buffers which fills up the heap memory. 

The ODataEntitySetIteratorResponseImpl.getBody call the getRawResponse() in the constructor call of 

entitySetIterator = new ClientEntitySetIterator<>(
                odataClient, getRawResponse(), ContentType.parse(getContentType()));
      }

 
However the constructor of ClientEntitySetIterator accepts InputStream.
So I do not understand the reason behind conversion of the payload into ByteArrayInputStream in the AbstractODataResponse. I am trying to figure the reason why this was done versus returning the payload InputStream as-is.

For fixing the problem, we have tried increasing the Java heap size but it is just a temporary solution since once the OData feed size increases further beyond a limit, it will fail again.

I also got heap dump for the Java crash and was able to visualize the largest object byte[] in a jprofiler to reach the same conclusion as above.