[HIVE-20203] Arrow SerDe leaks a DirectByteBuffer

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Eric Wohlstadter
• Reporter: Eric Wohlstadter
• Created: 18/Jul/18 18:46
• Updated: 17/Nov/22 08:54
• Resolved: 26/Jul/18 03:05

Description

ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task that uses the serde.

The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool.

This buffer is never closed and leaks about 1K of physical memory for each task.

This patch does three things:

1. Ensure the buffer is closed when the RecordWriter for the task is closed. 
2. Adds per-task memory accounting by assigning a ChildAllocator to each task from the RootAllocator.
3. Enforces that the ChildAllocator for a task has released all memory assigned to it, when the task is completed. 

The patch assumes that close() is always called on the RecordWriter when a task is finished (even if there is a failure during task execution).