[BIGTOP-1544] [BigPetStore] Use of java.io.File and check for directory existence will fail with non-POSIX DFSs (HDFS, S3, etc.)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: blueprints
• Assignee: R J Nowling
• Reporter: R J Nowling
• Created: 21/Nov/14 17:27
• Updated: 18/Mar/15 22:47
• Resolved: 22/Nov/14 01:35

Description

The BigPetStore Spark driver uses a java.io.File object to store the output directory path. The File's methods are used to check if the path exists and is a directory.

Since non-POSIX DFSs (S3, HDFS, etc.) will not have paths of the appropriate format and will not be accessible through a POSIX interface, these checks will fail, preventing the usage of S3, HDFS, etc.