[TINKERPOP-2454] OOM error when running gremlin queries asynchronously with JAVA

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 3.4.6, 3.4.8
• Component/s: driver
• Assignee: Unassigned
• Reporter: Vikas Yadav
• Created: 21/Oct/20 10:47
• Updated: 02/Nov/20 15:59
• Resolved: 02/Nov/20 15:59

Description

We have created a rest API that executes a gremlin query on the Janus graph and returns the result in JSON format. API works file for small result sets. But for large result sets, when we hit the API asynchronously, it gives the following error, (max heap size -Xmx4g

java.lang.OutOfMemoryError: GC overhead limit exceeded

I am using curl with & to hit API asynchronously,

_{curl --location --request GET 'http://HOST:PORT/graph/search?gremlin=query &
curl --location --request GET 'http://HOST:PORT/graph/search?gremlin=query &
curl --location --request GET 'http://HOST:PORT/graph/search?gremlin=query &
curl --location --request GET 'http://HOST:PORT/graph/search?gremlin=query &}

Code to connect to janus graph,

_{cluster = Cluster.open(config);
connect = cluster.connect();}

_{submit = connect.submit(gremlin);
Iterator<Result> resultIterator = submit.iterator();
int count=0;
while (resultIterator.hasNext()){
//add to list, commented to check OOM error
}}

 
Configurations,

_{config.setProperty("connectionPool.maxContentLength", "50000000");
config.setProperty("connectionPool.maxInProcessPerConnection", "30");
config.setProperty("connectionPool.maxInProcessPerConnection", "30");
config.setProperty("connectionPool.maxSize", "30");
config.setProperty("connectionPool.minSize", "1");
config.setProperty("connectionPool.resultIterationBatchSize", "200");}
 

Gremlin driver,
 _{org.apache.tinkerpop.gremlin-driver:3.4.6}
 

Query returns around 17K records with 80MB size.

How to handle a large resultset like a cursor so that not all the data is loaded in the memory?
Is there any configuration that I am missing?

From profiling, it is clear that the gremlin driver is causing the issue but I am not sure how to fix it and release the memory.

Please let me know if you need more details.

 

Thanks.