[SPARK-30474] Writing data to parquet with dynamic partitionOverwriteMode should not do the folder rename in commitjob stage

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Zaisheng Dai
• Created: 09/Jan/20 18:15
• Updated: 16/Mar/20 22:50

Description

In the current spark implementation if you set,

spark.sql.sources.partitionOverwriteMode=dynamic

even with 

mapreduce.fileoutputcommitter.algorithm.version=2

it would still rename the partition folder sequentially in commitJob stage as shown here: 

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala#L188

https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala#L184

 

This is very slow in cloud storage. We should commit the data similar to FileOutputCommitter v2?

 

Issue Links

• relates to: SPARK-20049 Writing data to Parquet with partitions takes very long after the job finishes