[ARROW-11569] Out-of-heap memory leaks in FlightClient.getStream

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: FlightRPC
• Labels: FlightClient MemoryLeak
• Environment: Java version: 8 ; Flight&Arrow version: 3.0.0
• External issue URL: https://github.com/apache/arrow/issues/27439
• Assignee: Unassigned
• Reporter: lulijun
• Created: 09/Feb/21 07:50
• Updated: 11/Jan/23 08:20

Description

I'm trying to use Arrow and Flight in my Java project, I used FlightClient.getStream to request data from a remote node, but I found the allocator's allocated memory size grows with the request.

Java version: 8

Flight&Arrow version: 3.0.0

 

Here's a snippet of my code, I found the allocated memory of flithStream still exist after root.close:

try (FlightStream flightStream = client.getStream(ticket, CallOptions.timeout(1, TimeUnit.MILLISECONDS))) {
    VectorSchemaRoot root = flightStream.getRoot();
    root.clear();
    while (flightStream.next())

{         // transfer root to ArrowRecordBatch and execute         dispatchNext(shuffleChunk, root);         root.clear();     }

    root.close();
} catch (FlightRuntimeException fe) {
    String errorMsg = "IndexReadOperator read index fail ={}  "+indexChunkDesc.toString();
    LOGGER.error(errorMsg, fe);
    throw new NetworkException(errorMsg, fe);
} catch (Exception e)

{     LOGGER.error("IndexReadOperator read index fail!", e); throw new NetworkException("IndexReadOperator getStream fail! ", e); }

 

Is this a bug or am I using it incorrectly?