[SPARK-17861] Store data source partitions in metastore and push partition pruning into metastore

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: SQL
• Target Version/s: 2.1.0
• Assignee: Eric Liang
• Reporter: Reynold Xin
• Created: 11/Oct/16 00:57
• Updated: 01/Aug/18 18:11
• Resolved: 30/Nov/16 04:06

Description

Initially, Spark SQL does not store any partition information in the catalog for data source tables, because initially it was designed to work with arbitrary files. This, however, has a few issues for catalog tables:

1. Listing partitions for a large table (with millions of partitions) can be very slow during cold start.
2. Does not support heterogeneous partition naming schemes.
3. Cannot leverage pushing partition pruning into the metastore.

This ticket tracks the work required to push the tracking of partitions into the metastore. This change should be feature flagged.

Issue Links

• is depended upon by: HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries
• is related to: SPARK-15691 Refactor and improve Hive support
• relates to: SPARK-16980 Load only catalog table partition metadata required to answer a query

Sub-Tasks

• Load only catalog table partition metadata required to answer a query: Resolved
• Feature flag SPARK-16980: Resolved
• Refactor FileCatalog classes to simplify the inheritance tree: Resolved
• Fix refreshByPath for converted Hive tables: Resolved
• Enable metastore partition pruning for unconverted hive tables by default: Resolved
• Add back a file status cache for catalog tables: Resolved
• should not always lowercase partition columns of partition spec in parser: Resolved
• Use metastore for managing filesource table partitions as well: Resolved
• put hive serde table schema to table properties like data source table: Resolved
• Can't filter over mixed case parquet columns of converted Hive tables: Resolved
• Optimize insert to not require REPAIR TABLE: Resolved
• ExternalCatalogSuite should test with mixed case fields: Resolved
• Avoid using Union to chain together create table and repair partition commands: Resolved
• INSERT OVERWRITE TABLE ... PARTITION will overwrite the entire Datasource table instead of just the specified partition: Resolved
• INSERT [INTO|OVERWRITE] TABLE ... PARTITION for Datasource tables cannot handle partitions with custom locations: Resolved
• data source tables should support truncating partition: Resolved
• HiveClient.getPartitionsByFilter throws an exception for some unsupported filters when hive.metastore.try.direct.sql=false: Resolved
• Rename partitionProviderIsHive -> tracksPartitionsInCatalog: Resolved
• ALTER TABLE ... ADD PARTITION does not play nice with mixed-case partition column names: Resolved
• correct several partition related behaviours of ExternalCatalog: Resolved
• Revert hacks in parquet and orc reader to support case insensitive resolution: Resolved
• Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions: Resolved
• Update documentation for hive partition management in 2.1: Resolved
• Append with df.saveAsTable writes data to wrong location: Resolved
• Major performance regression in SHOW PARTITIONS on partitioned Hive tables: Resolved
• Verify number of hive client RPCs in PartitionedTablePerfStatsSuite: Resolved
• Partition name/values not escaped correctly in some cases: Resolved
• Regression in file listing performance: Resolved
• Incorrect behaviors in overwrite table for datasource tables: Resolved
• Creating a partitioned datasource table should not scan all files for table: Resolved
• Return Nothing when Querying a Partitioned Data Source Table without Repairing it: Closed