Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29043

[History Server]Only one replay thread of FsHistoryProvider work because of straggler

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.4
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      As shown in the attachment, we set spark.history.fs.numReplayThreads=30 for spark history server.
      However, there is only one replay thread work because of straggler.

      Let's check the code.
      https://github.com/apache/spark/blob/7f36cd2aa5e066a807d498b8c51645b136f08a75/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L509-L547

      There is a synchronous operation for all replay tasks.

        Attachments

          Activity

            People

            • Assignee:
              hzfeiwang feiwang
              Reporter:
              hzfeiwang feiwang

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment