Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28302

SparkLauncher: The process cannot access the file because it is being used by another process

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.3.4, 2.4.4, 3.0.0
    • Component/s: Spark Core
    • Labels:
      None
    • Environment:

      Windows 7

    • Flags:
      Patch

      Description

      When using SparkLauncher to submit applications concurrently with a thread pool under Windows, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. (Issue can be reproduced with attach file.)

       

      After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it  instantly( < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it try to write the same file which has already been occupied by another app.

       

       

        Attachments

          Activity

            People

            • Assignee:
              Ngone51 wuyi
              Reporter:
              Ngone51 wuyi

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Issue deployment