Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21223

Thread-safety issue in FsHistoryProvider

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Currently, Spark HistoryServer use a HashMap named fileToAppInfo in class FsHistoryProvider to store the map of eventlog path and attemptInfo.
      When use ThreadPool to Replay the log files in the list and merge the list of old applications with new ones, multi thread may update fileToAppInfo at the same time, which may cause Thread-safety issues.

      for (file <- logInfos) {
             tasks += replayExecutor.submit(new Runnable {
              override def run(): Unit = mergeApplicationListing(file)
           })
       }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gostop_zlx zenglinxi
                Reporter:
                gostop_zlx zenglinxi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: