[IMPALA-2523] Make HdfsTableSink aware of clustered input

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.2, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: performance ramp-up
• Epic Link: Improve the reliability and effectiveness of ETL
• Docs Text: This needs to be documented
• Target Version: Impala 2.8.0
• Assignee: Lars Volker
• Reporter: Mostafa Mokhtar
• Created: 09/Oct/15 19:07
• Updated: 20/Mar/17 22:27
• Resolved: 22/Nov/16 10:32

Description

The HdfsParquetTableWriter needs to be aware that incoming data is clustered if the corresponding insert statement has a "clustered" hint. Only a single open partition should be maintained and flushed by calling FinalizePartitionFile() when the partition-key values change.

For now, no changes should be made to other table sinks.

Issue Links

• is blocked by: IMPALA-2521 Introduce CLUSTERED plan hint for insert statements