[PARQUET-787] Add a size limit for heap allocations when reading

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.10.0
• Component/s: parquet-mr
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 01/Dec/16 20:49
• Updated: 21/Feb/18 17:41
• Resolved: 21/Feb/18 17:41

Description

G1GC allocates humongous objects directly in the old generation to avoid unnecessary copies, which means that these allocations aren't garbage collected until a full GC runs. Humongous objects are objects that are 50% of the region size or more. Region size is at most 32MB (see the table for region size from heap size).

Parquet currently allocates a huge buffer for each contiguous group of column chunks, which in many cases is not garbage collected until a full GC. Adding a size limit for the allocation size should allow users to break row groups across multiple buffers so that buffers get collected when they have been read.

Issue Links

• blocks: PARQUET-1189 Release Parquet Java 1.10
• links to: Pull Request 390