[IMPALA-9477] Only add required partitions to TDescriptorTable for hdfs table sinks

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 3.3.0
• Component/s: Frontend
• Epic Color: ghx-label-12
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 07/Mar/20 00:37
• Updated: 07/Mar/20 00:38

Description

In https://github.com/apache/impala/blob/b068c2c/fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java#L196 we fall back to adding all partition metadata for the target table for DMLs.

I don't think this is necessary if query specifies a target partition. Or if it has multiple partition keys and it specified constants for some of them.

If a table has a large number of partitions, the memory overhead can be very high because of things like IMPALA-8835

Issue Links

• relates to: IMPALA-8835 Reduce unnecessary memory usage for Thrift structures