[HIVE-7685] Parquet memory manager

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Serializers/Deserializers
• Labels: TODOC15
• Assignee: Dong Chen
• Reporter: Brock Noland
• Created: 12/Aug/14 05:33
• Updated: 18/Aug/15 23:14
• Resolved: 06/Jan/15 01:19

Description

Similar to HIVE-4248, Parquet tries to write large very large "row groups". This causes Hive to run out of memory during dynamic partitions when a reducer may have many Parquet files open at a given time.

As such, we should implement a memory manager which ensures that we don't run out of memory due to writing too many row groups within a single JVM.

Issue Links

• blocks: HIVE-8120 Umbrella JIRA tracking Parquet improvements
• is related to: HIVE-11598 Document Configuration for Parquet Files
• relates to: PARQUET-164 Warn when parquet memory manager kicks in
• relates to: HIVE-10149 Shuffle Hive data before storing in Parquet
• relates to: PARQUET-108 Parquet Memory Management in Java
• relates to: PARQUET-177 MemoryManager ensure minimum Column Chunk size
• links to: Parquet Documentation