Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28302

SparkLauncher: The process cannot access the file because it is being used by another process

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.3.4, 2.4.4, 3.0.0
    • Spark Core
    • None
    • Windows 7

    • Patch

    Description

      When using SparkLauncher to submit applications concurrently with a thread pool under Windows, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. (Issue can be reproduced with attach file.)

       

      After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it  instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app.

       

       

      Attachments

        1. Main.scala
          1 kB
          wuyi

        Issue Links

          Activity

            People

              Ngone51 wuyi
              Ngone51 wuyi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: