[PARQUET-177] MemoryManager ensure minimum Column Chunk size

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.0
• Component/s: parquet-mr
• Assignee: Daniel Weeks
• Reporter: Daniel Weeks
• Created: 04/Feb/15 00:56
• Updated: 23/Jun/24 03:27
• Resolved: 05/Feb/15 22:37

Description

The memory manager currently has no limit to how small it will make row groups. This is problematic because jobs that have a large number of writers can result in tiny row groups that hurt performance.

The following patch will allow a configurable minimum size before killing the job. Default is currently no limit.

Issue Links

• is related to: HIVE-7685 Parquet memory manager