[FLINK-14249] Complete partition support for batch sql

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: Table SQL / Planner
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 27/Sep/19 10:43
• Updated: 15/Apr/22 09:25
• Resolved: 30/Dec/19 08:10

Description

A partition is a division of a logical database or its constituent elements into distinct independent parts. Database partitioning is normally done for manageability, performance or availability reasons, or for load balancing.

Partition is widely used in hive. Especially in the ETL domain, most tables have partition attributes, which allow users to continue processing. Partition is more convenient for data management, time partitioning and business partitioning are common.

We have supported a basic version for Flink table partitioning at 1.9 release, and this JIRA aims to improve partitioning support.

See https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support

Issue Links

• relates to: FLINK-27237 Partitioned table statement enhancement
• relates to: FLINK-14256 [Umbrella] Introduce FileSystemTableFactory with partitioned support

Sub-Tasks

• Add hash distribution and sort grouping only when dynamic partition insert: Closed
• Introduce FileSystemOutputFormat for batch: Closed
• Don't use ContinuousFileReaderOperator to support multiple paths: Resolved
• Partition field names should be got from CatalogTable instead of source/sink: Closed
• Introduce listPartitionsByFilter to Catalog: Closed
• Implement listPartitionsByFilter to HiveCatalog: Closed
• Handle escape characters for partition path: Closed
• Enable partition statistics in blink planner: Closed