[HUDI-5289] WriteStatus RDD is recalculated in cluster

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.1, 0.12.3
• Component/s: spark
• Labels: pull-request-available
• Assignee: Xinyu Zou
• Reporter: Xinyu Zou
• Created: 29/Nov/22 02:24
• Updated: 30/Mar/23 02:23
• Resolved: 30/Mar/23 02:23

Description

Step:

spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
--conf spark.driver.memory=40G \
--conf spark.executor.instances=20 \
--conf spark.executor.memory=40G \
--conf spark.executor.cores=4 \
hudi-utilities-bundle_2.11-0.12.0.jar \
--props clusteringjob.properties \
--mode scheduleAndExecute \
--base-path xxx \
--table-name xxx \
--spark-memory 40g 

The following are the two stages about the job, they are all related to the calculation of WriteStatus, but some tasks in stage96 have been recalculated which taking more than ten minutes

here is stage 65

here is stage 96

Issue Links

• links to: GitHub Pull Request #7373
• links to: GitHub Pull Request #8275