[SPARK-30190] HistoryServerDiskManager will fail on appStoreDir in s3

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: thierry accart
• Created: 09/Dec/19 13:42
• Updated: 16/Mar/20 22:53

Description

Hi

While setting spark.eventLog.dir to s3a://... I realized that it requires destination directory to preexists for S3

This is explained I think in HistoryServerDiskManager's appStoreDir: it tries check if directory exists or can be created

if (!appStoreDir.isDirectory() && !appStoreDir.mkdir()) {throw new IllegalArgumentException(s"Failed to create app directory ($appStoreDir).")}

But in S3, a directory does not exists and cannot be created: directories don't exists by themselves, they are only materialized due to existence of objects.

Before proposing a patch, I wanted to know what are the prefered options :
should we have a spark option to skip the appStoreDir test, or skip it only when a particular scheme is set, have a custom implementation of HistoryServerDiskManager ...?

 

Note for people facing the IllegalArgumentException: Failed to create app directory you just have to put an empty file in bucket destination 'path'.