[SPARK-5923] Very slow query when using Oracle hive metastore and table has lots of partitions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Matthew Taylor
• Created: 20/Feb/15 12:33
• Updated: 07/Oct/16 07:54
• Resolved: 07/Oct/16 07:54

Description

This has two aspects

• The direct sql support for oracle is broken in hive 0.13.1. Fails when partitions get bigger than 1000 due oracle limitation on IN clause. This cause fall back to ORM which is very slow(20 minutes to even start the query)
• Hive it self does not suffer this problem as it passes down to the metadata query, filter terms that restrict the partitions returned. SparkSQL is always asking for all partitions event if they are not all needed. Even when we patched hive it was still taking 2 minutes