[HIVE-4157] ORC runs out of heap when writing

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0
• Fix Version/s: 0.11.0
• Component/s: Serializers/Deserializers
• Assignee: Kevin Wilfong
• Reporter: Kevin Wilfong
• Created: 12/Mar/13 23:10
• Updated: 16/May/13 21:10
• Resolved: 29/Mar/13 00:27

Description

The OutStream class used by the ORC file format seems to aggressively allocate memory for ByteBuffers and doesn't seem too eager to give it back.

This causes issues with heap space, particularly when a wide tables/dynamic partitions are involved.

As a first step to resolving this problem, the OutStream class can be modified to lazily allocate memory, and more actively make it available for garbage collection.

Follow ups could include checking the amount of free memory as part of determining if a spill is needed.