[MAPREDUCE-7185] Parallelize part files move in FileOutputCommitter

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0, 2.9.2
• Target Version/s: 3.5.0
• Assignee: Igor Dvorzhak
• Reporter: Igor Dvorzhak
• Created: 11/Feb/19 22:51
• Updated: 04/Jan/24 09:33

Description

If map task outputs multiple files it could be slow to move them from temp directory to output directory in object stores (GCS, S3, etc).

To improve performance we need to parallelize move of more than 1 file in FileOutputCommitter.

Repro:
Start spark-shell:

spark-shell --num-executors 2 --executor-memory 10G --executor-cores 4 --conf spark.dynamicAllocation.maxExecutors=2

From spark-shell:

val df = (1 to 10000).toList.toDF("value").withColumn("p", $"value" % 10).repartition(50)
df.write.partitionBy("p").mode("overwrite").format("parquet").options(Map("path" -> s"gs://some/path")).saveAsTable("parquet_partitioned_bench")

With the fix execution time reduces from 130 seconds to 50 seconds.

Issue Links

• relates to: MAPREDUCE-7267 During commitJob, enable merge paths with multi threads