[SPARK-2119] Reading Parquet InputSplits dominates query execution time when reading off S3

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: SQL
• Target Version/s: 1.1.0
• Assignee: Cheng Lian
• Reporter: Michael Armbrust
• Created: 12/Jun/14 01:38
• Updated: 17/Jul/14 08:17
• Resolved: 16/Jul/14 16:46

Description

Here's the relevant stack trace where things are hanging:

	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:326)
	at parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:370)
	at parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:344)
	at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:90)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:201)

We should parallelize or cache or something here.

Issue Links

• is related to: SPARK-2551 Cleanup FilteringParquetRowInputFormat
• relates to: PARQUET-16 Unnecessary getFileStatus() calls on all part-files in ParquetInputFormat.getSplits