[SPARK-6351] ParquetRelation2 does not support paths for different file systems

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.1
• Component/s: SQL
• Labels: backport-needed
• Target Version/s: 1.3.1
• Assignee: Volodymyr Lyubinets
• Reporter: Pei-Lun Lee
• Created: 16/Mar/15 07:45
• Updated: 16/Mar/15 19:19
• Resolved: 16/Mar/15 19:19

Description

With new parquet API, if multiple parquet paths with different file systems (e.g. s3n and hdfs) are given, the error "IllegalArgumentException: Wrong FS" will be raised. Due to the FileSystem is created from hadoop configuration, which only allow one default file system. This behavior was supposed to work in old API.

Issue Links

• duplicates: SPARK-6330 newParquetRelation gets incorrect FileSystem
• links to: [Github] Pull Request #5039 (ypcat)