[HIVE-2087] Dynamic partition insert performance problem

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.0
• Component/s: Metastore

Environment

Amazon EMR, S3

• Tags: dynamic partition peformace
• Assignee: Unassigned
• Reporter: Q Long
• Created: 31/Mar/11 18:22
• Updated: 09/May/11 18:04

Description

Create an external(backed by S3) table T, make it partitioned by column P. Populate table T so it has large number of partitions (say 100). Execute statement like

insert overwrite table T partition (p) select * from another_table

check hive server log, and it will show that all existing partitions will be read and loaded before any mapper starts working. This feels excessive, given that the insert statement may only create or overwrite a very small number of partitions. Is there other reason that insert using dynamic partition requires loading the whole table?