[SPARK-18931] Create empty staging directory in partitioned table on insert

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.2
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Egor Pahomov
• Created: 20/Dec/16 00:03
• Updated: 12/Dec/22 18:10
• Resolved: 28/Dec/16 04:28

Description

CREATE TABLE temp.test_partitioning_4 (
num string
)
PARTITIONED BY (
day string)
stored as parquet

On every

INSERT INTO TABLE temp.test_partitioning_4 PARTITION (day)
select day, count as num from
hss.session where year=2016 and month=4
group by day

new directory ".hive-staging_hive_2016-12-19_15-55-11_298_3412488541559534475-4" created on HDFS. It's big issue, because I insert every day and bunch of empty dirs on HDFS is very bad for HDFS.

Issue Links

• is duplicated by: SPARK-18703 Insertion/CTAS against Hive Tables: Staging Directories and Data Files Not Dropped Until Normal Termination of JVM