[DRILL-6552] Drill Metadata management "Drill Metastore"

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.18.0
• Component/s: Metadata
• Labels: doc-impacting
• Assignee: Vova Vysotskyi
• Reporter: Vitalii Diravka
• Created: 28/Jun/18 16:46
• Updated: 01/Apr/20 17:43
• Resolved: 20/Mar/20 10:44

Description

It would be useful for Drill to have some sort of metastore which would enable Drill to remember previously defined schemata so Drill doesn’t have to do the same work over and over again.

It allows to store schema and statistics, which will allow to accelerate queries validation, planning and execution time. Also it increases stability of Drill and allows to avoid different kind if issues: "schema change Exceptions", "limit 0" optimization and so on. 

One of the main candidates is Hive Metastore.
Starting from 3.0 version Hive Metastore can be the separate service from Hive server:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administration

Optional enhancement is storing Drill's profiles, UDFs, plugins configs in some kind of metastore as well.

Issue Links

• incorporates: DRILL-6852 Adapt current Parquet Metadata cache implementation to use Drill Metastore API
• incorporates: DRILL-7098 File Metadata Metastore Plugin
• is depended upon by: DRILL-5192 REFRESH table METADATA as default system option
• is related to: DRILL-7028 Reduce the planning time of queries on large Parquet tables with large metadata cache files
• is related to: DRILL-7430 Drill Metastore analyze improvements
• is related to: DRILL-7684 Implement Drill Metastore metadata usage for Hive storage plugin
• relates to: DRILL-3588 Write back to Hive Metastore
• relates to: DRILL-6035 Specify Drill's JSON behavior
• relates to: DRILL-1328 Support table statistics
• requires: DRILL-5603 Replace String file paths to Hadoop Path
• requires: DRILL-6604 Upgrade Drill Hive client to Hive3.1 version
• links to: Design document
• links to: Hangout Presentation
• links to: Metastore metadata collecting and retrieval

Sub-Tasks

• Research and investigate a way for collecting and storing table statistics in the scope of metastore integration: Resolved
• Adapt current Parquet Metadata cache implementation to use Drill Metastore API: Resolved
• Implement caching of BaseMetadata classes: Resolved
• File Metadata Metastore Plugin: Resolved
• Adapt statistics to use Drill Metastore API: Resolved
• Refactor Metadata interfaces and classes to contain all needed information for the File based Metastore: Resolved
• Move schema-related classes from exec module to be able to use them in metastore module: Resolved
• Implement Drill Iceberg Metastore plugin: Resolved
• Support Iceberg metadata expiration: Resolved
• Add vararg UDFs support: Resolved
• Introduce session options for the Drill Metastore: Resolved
• Create operator for handling metadata: Resolved
• Implement metadata usage for Parquet format plugin: Resolved
• Expose Drill Metastore data through INFORMATION_SCHEMA: Closed
• Implement metadata usage for text format plugin: Resolved
• Introduce ANALYZE TABLE statements: Resolved
• Allow passing table function parameters into ANALYZE statement: Resolved