[SPARK-29043] [History Server]Only one replay thread of FsHistoryProvider work because of straggler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.4
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: feiwang
• Reporter: feiwang
• Created: 11/Sep/19 01:56
• Updated: 02/May/22 05:27
• Resolved: 16/Dec/19 22:46

Description

As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for spark history server.
However, there is only one replay thread work because of straggler.

Let's check the code.
https://github.com/apache/spark/blob/7f36cd2aa5e066a807d498b8c51645b136f08a75/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L509-L547

There is a synchronous operation for all replay tasks.

Issue Links

• is related to: SPARK-39083 Fix FsHistoryProvider race condition between update and clean app data
• links to: GitHub Pull Request #25797