[FLINK-12755] implement HiveCatalog

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: Bowen Li
• Reporter: Bowen Li
• Created: 05/Jun/19 21:50
• Updated: 14/Nov/19 20:59
• Resolved: 14/Nov/19 20:59

Description

An umbrella JIRA for HiveCatalog.

Background: Previously we have GenericHiveMetastoreCatalog and HiveCatalog, and we decided to unify them into a single HiveCatalog with two code paths to deal with generic Flink metadata and Hive metadata.

This umbrella JIRA is for anything that is shared between the two code paths.

JIRA tickets related to one code path should be subtasks of FLINK-11479 or FLINK-11482

Issue Links

• is a parent of: FLINK-11479 support Hive metadata in HiveCatalog
• is a parent of: FLINK-11482 support generic Flink metadata in HiveCatalog
• Is contained by: FLINK-11275 Unified Catalog APIs
• links to: GitHub Pull Request #9151

Sub-Tasks

• Support mapping BinaryType, VarBinaryType, CharType, VarCharType, and DecimalType between Flink and Hive in HiveCatalog: Closed
• migrate HiveCatalog from TypeInformation-based old type system to DataType-based new type system: Closed
• Support ConnectorCatalogTable in HiveCatalog: Closed
• Support mapping ARRAY, MAP, ROW (STRUCT) between Flink and Hive in HiveCatalog: Closed
• use Flink's LogicalTypeRoot for type comparison in HiveTypeUtil: Closed
• check certain properties that are not supported when alter database in HiveCatalog: Closed
• add hadoop dependencies as 'flink-shaded-hadoop-2-uber' for flink-connector-hive to connect to remote hive metastore service: Closed
• package flink-connector-hive into flink distribution: Closed
• rename hive-site-path to hive-conf-dir to be consistent with standard name in Hive: Closed
• append hive-site.xml to path of Hive conf dir: Closed
• override default hadoop version from 2.4.1 to 2.7.5 in flink-connector-hive: Closed
• unify configs for meta-objects in HiveCatalog: Closed
• write documentation and quickstart for Flink-Hive integration: Closed
• move tests for data type mappings between Flink and Hive into its own test class: Closed
• add IT case for reading and writing generic table metadata via HiveCatalog: Closed