[SPARK-16188] Spark sql create a lot of small files

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: 1.6.0
• Component/s: SQL

Environment

spark 1.6.1

• Assignee: Unassigned
• Reporter: cen yuhai
• Created: 24/Jun/16 08:41
• Updated: 12/Dec/22 18:11
• Resolved: 29/Jun/16 00:55

Description

I find that spark sql will create files as many as partition size. When the results are small, there will be too many small files and most of them are empty.

Hive have a function to detect the avg of file size. If avg file size is smaller than "hive.merge.smallfiles.avgsize", hive will add a job to merge files.