Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28302

SparkLauncher: The process cannot access the file because it is being used by another process

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.3.4, 2.4.4, 3.0.0
    • Component/s: Spark Core
    • Labels:
      None
    • Environment:

      Windows 7

    • Flags:
      Patch

      Description

      When using SparkLauncher to submit applications concurrently with a thread pool under Windows, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. (Issue can be reproduced with attach file.)

       

      After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it  instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app.

       

       

        Attachments

        1. Main.scala
          1 kB
          wuyi

          Issue Links

            Activity

              People

              • Assignee:
                Ngone51 wuyi
                Reporter:
                Ngone51 wuyi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: