[DRILL-6852] Adapt current Parquet Metadata cache implementation to use Drill Metastore API

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Labels: ready-to-commit
• Assignee: Vova Vysotskyi
• Reporter: Vova Vysotskyi
• Reviewer: Aman Sinha
• Created: 14/Nov/18 18:13
• Updated: 21/Mar/19 17:34
• Resolved: 21/Mar/19 17:34

Description

According to the design document for DRILL-6552, existing metadata cache API should be adapted to use generalized API for metastore and parquet metadata cache will be presented as the implementation of metastore API.

The aim of this Jira is to refactor Parquet Metadata cache implementation and adapt it to use Drill Metastore API.

Execution plan:

• Refactor AbstractParquetGroupScan and its implementations to use metastore metadata classes. Store Drill data types in metadata files for Parquet tables.
• Storing the least restrictive type instead of current first file’s column data type.
• Rework logic in AbstractParquetGroupScan to allow filtering at different metadata layers: partition, file, row group, etc. The same for pushing the limit.
• Implement logic to convert existing parquet metadata to metastore metadata to preserve backward compatibility.
• Implement fetching metadata only when it is needed (for filtering, limit, count etc.)

Issue Links

• is part of: DRILL-6552 Drill Metadata management "Drill Metastore"
• links to: Class diagram
• links to: GitHub Pull Request #1646