[ARROW-16163] [Go] IPC FileReader leaks memory when used with ZSTD compression

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: Go
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/31569
• Assignee: Nicolas MOREAU
• Reporter: Nicolas MOREAU
• Created: 11/Apr/22 14:05
• Updated: 11/Jan/23 11:42
• Resolved: 13/Apr/22 17:15

Description

I'll submit a PR

 

When using the IPC Reader to read data compressed in zstd, we observed what looked like a memory leak.

After further debugging, we observed that calling the IPC Reader created goroutines that were never stopped, and consumed memory.

 

After deeper investigation, we found the following root cause:

 

In arrow/ipc/file_reader.go, the newRecord function calls getDecompressor, that returns a zstd.Decoder object.

 

When used in its stream version (calling .Read()), this decoder makes use of a Goroutine with a channel to receive input. This channel is closed by the decoder's .Close() function, which ends the Goroutine and therefore the collection of its memory by the GC.

 

After using the zstd Decoder, the IPC FileReader should close it to prevent the leak.

Issue Links

• links to: GitHub Pull Request #12857