Description
As of current 2.1, INSERT OVERWRITE with dynamic partitions against a Datasource table will overwrite the entire table instead of only the updated partitions as in Hive. It also doesn't respect custom partition locations.
We should delete only the proper partitions, scan the metastore for affected partitions with custom locations, and ensure that deletes/writes go to the right locations for those as well.
Attachments
Issue Links
- relates to
-
SPARK-18183 INSERT OVERWRITE TABLE ... PARTITION will overwrite the entire Datasource table instead of just the specified partition
-
- Resolved
-
- links to
