[SPARK-10216] Avoid creating empty files during overwrite into Hive table with group by query

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Hyukjin Kwon
• Reporter: Keuntae Park
• Created: 25/Aug/15 07:04
• Updated: 12/Dec/22 21:10
• Resolved: 19/Jul/17 16:41

Description

Exchange from GROUP BY query results in at least certain amount of partitions specified in 'spark.sql.shuffle.partition'.
Hence, even when the number of distinct group-by key is small,
INSERT INTO with GROUP BY query try to make at least 200 files (default value of 'spark.sql.shuffle.partition'),
which results in lots of empty files.
I think it is undesirable because upcoming queries on the resulting table will also make zero size partitions and unnecessary tasks do nothing on handling the queries.

Issue Links

• breaks: SPARK-15393 Writing empty Dataframes doesn't save any _metadata files
• is duplicated by: SPARK-15393 Writing empty Dataframes doesn't save any _metadata files
• is duplicated by: SPARK-21105 Useless empty files in hive table
• relates to: SPARK-21435 Empty files should be skipped while write to file
• links to: [Github] Pull Request #8411 (sirpkt)
• links to: [Github] Pull Request #12855 (HyukjinKwon)
• links to: [Github] Pull Request #13181 (marmbrus)
• links to: [Github] Pull Request #18654 (xuanyuanking)