[FLINK-14133] Improve batch sql and hive integrate performance milestone-1

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: Connectors / Hive, Table SQL / Planner
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 20/Sep/19 01:56
• Updated: 09/Dec/19 08:26
• Resolved: 09/Dec/19 08:26

Description

Now we have basically merged the batch function of blink planner and basically integrated hive. But there are still many problems with performance, and we need to ensure basic performance.

Issue Links

• is a parent of: FLINK-14063 Operators use fractions to decide how many managed memory to allocate
• relates to: FLINK-14799 Improve batch sql and hive integrate performance milestone-2

Sub-Tasks

• HiveTableSource should implement ProjectableTableSource: Closed
• Introduce LimitableTableSource to optimize limit: Resolved
• Introduce vectorized orc InputFormat for blink runtime: Closed
• HiveTableSource should implement LimitableTableSource interface: Closed
• Join condition could be simplified in logical phase: Closed
• blink planner should also fetch catalog statistics for permanent CatalogTableImpl: Closed
• Optimize mapred.HadoopInputSplit to not serialize conf when split is not configurable: Closed
• Decide column nullability according to Hive constraints: Closed
• Improve batch schedule check input consumable performance: Closed
• Optimize BaseRowSerializer to use Projection instead of switch: Closed
• extends max/min type in ColumnStats from Number to Comparable: Closed
• Introduce parallelism inference for HiveSource: Closed
• Introduce data compression to blocking shuffle.: Closed
• add local aggregate to solve data skew for ROLLUP/CUBE case: Closed