[SPARK-6904] SparkSql - HiveContext - optimize reading partition data from metastore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 14/Apr/15 16:01
• Updated: 14/Apr/15 18:14
• Resolved: 14/Apr/15 18:14

Description

I was trying out spark sql using the HiveContext and doing a select on a partitioned table with lots of partitions (16,000+). It took over 6 minutes before it even started the job. It looks like it was querying the Hive metastore and got a good chunk of data back. Which I'm guessing is info on the partitions. Running the same query using hive takes 45 seconds for the entire job.

It would be nice if we could optimize on the partitions when reading from the metastore.

Issue Links

• duplicates: SPARK-6910 Support for pushing predicates down to metastore for partition pruning