[FLINK-14256] [Umbrella] Introduce FileSystemTableFactory with partitioned support

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.11.0
• Component/s: Table SQL / Planner
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 27/Sep/19 15:02
• Updated: 01/Feb/21 06:29
• Resolved: 04/Jun/20 08:31

Description

Introduce FileSystemTableFactory to unify all file system connectors.

More information in https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table

Issue Links

• is related to: FLINK-17778 [umbrella] FileSystemTableFactory improve
• is related to: FLINK-14249 Complete partition support for batch sql

Sub-Tasks

• Integrate hive to streaming file sink: Closed
• Integrate csv to FileSystemTableFactory: Closed
• Integrate file system connector to streaming sink: Closed
• Introduce Row Csv InputFormat: Resolved
• Introduce Row Csv Encoder: Closed
• Support partition for temporary table: Closed
• Add option to close shuffle when dynamic partition inserting: Closed
• Support partition prune by getPartitions of source: Resolved
• Expose partitioned by grammar: Resolved
• Introduce FileSystemTableFactory, FileSystemTableSource, FileSystemTableSink: Resolved
• SingleDirectoryWriter should not produce file when no input record: Resolved
• Introduce table row write support for parquet writer: Resolved
• Integrate parquet to file system connector: Resolved
• Add docs for FileSystem connector: Closed
• Add isBounded to TableSinkFactory#Context: Resolved
• Integrate json to file system connector: Closed
• Integrate orc to file system connector: Closed
• sink.rolling-policy.time.interval default value should be bigger: Closed
• Move FileSystemFormatFactory to table common: Closed
• Create sql-jars for parquet and orc: Closed
• Filesystem streaming sink support partition commit (success file): Closed
• Add project ITCase for partition table in filesystem connector: Closed
• Integrate avro to file system connector: Closed
• Add documentation for ORC format: Closed
• Add documentation for Parquet format: Closed
• Add filesystem documentation for Avro: Closed