[IMPALA-257] Parquet writer uses excessive memory with partitions

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 0.7
• Fix Version/s: Impala 1.1
• Assignee: Nong Li
• Reporter: Nong Li
• Created: 15/Apr/13 21:19
• Updated: 22/Jul/13 21:33
• Resolved: 21/Jul/13 20:06

Description

The parquet table writer uses a lot of memory and this grows linearly with the number of output partitions. We'd like to write large files (~512MB-1GB) and these need to be buffered per partition. If the output has 100 partitions, this is 50GB+ RAM that is required.

The buffering will be resolved if we can write multiple columns to different hdfs files, in which case we don't need to buffer at all.

An alternative solution is to write the cols to local disk and then at the end stitch up the files.