[SPARK-46215] Improve `FileSystemPersistenceEngine` to allow non-exist parents

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 02/Dec/23 22:24
• Updated: 11/Dec/23 02:46
• Resolved: 03/Dec/23 04:24

Description

To prevent the following

info]   java.io.IOException: No such file or directory
[info]   at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
[info]   at java.base/java.io.File.createNewFile(File.java:1043)
[info]   at org.apache.spark.deploy.master.FileSystemPersistenceEngine.serializeIntoFile(FileSystemPersistenceEngine.scala:62)
[info]   at org.apache.spark.deploy.master.FileSystemPersistenceEngine.persist(FileSystemPersistenceEngine.scala:45)

Issue Links

• links to: GitHub Pull Request #44127
• links to: GitHub Pull Request #44285