[IMPALA-5707] HDFS table sinks should operate within a memory constraint

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: resource-management
• Epic Link: Operator memory constraints
• Target Version: Product Backlog
• Epic Color: ghx-label-3
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 24/Jul/17 18:24
• Updated: 05/Feb/18 23:03

Description

We should modify the Parquet table sink so that it reserves memory upfront and allocates the bulk of its memory (column buffers, etc) from the buffer pool.

The reservation calculation should be fairly straightforward for clustered inserts or single-partition inserts, but currently the memory consumption of dynamically partitioned non-cluster inserts is not bounded. In that case we would need to flush partitions to disk to free memory.

One possibly tricky edge case is inserting large string values, since the values could be larger than the regular 2MB buffer size.

Issue Links

• is related to: IMPALA-5293 Turn insert clustering on by default