[SPARK-10673] spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.4.0, 1.5.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Miklos Christine
• Created: 17/Sep/15 21:50
• Updated: 20/Oct/16 12:10
• Resolved: 10/Oct/16 05:40

Description

In Spark 1.4, spark.sql.hive.verifyPartitionPath was set to true by default.

In Spark 1.5, it is now set to false by default.

If a table has a lot of partitions in the underlying filesystem, the code unnecessarily checks for all the underlying directories when executing a query.
https://github.com/apache/spark/blob/v1.5.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala#L162

Structure:

/user/hive/warehouse/table1/year=2015/month=01/
/user/hive/warehouse/table1/year=2015/month=02/
/user/hive/warehouse/table1/year=2015/month=03/
...
/user/hive/warehouse/table1/year=2014/month=01/
/user/hive/warehouse/table1/year=2014/month=02/

If the registered partitions only contain year=2015 when you run "show partitions table1", this code path checks for all directories under the table's root directory. This incurs a significant performance penalty if there are a lot of partition directories.