Details
-
New Feature
-
Status: Reopened
-
Not a Priority
-
Resolution: Unresolved
-
None
-
None
Description
Vectorization is a popular technique in SQL engines today. Compared with traditional row-based approach, it has some distinct advantages, for example:
- Better use of CPU resources (e.g. SIMD)
- More compact memory layout
- More friendly to compressed data format.
Currently, Flink is based on a row-based SQL engine for both stream and batch workloads. To enjoy the above benefits, we want to bring vectorization to Flink. This involves substantial changes to the existing code base. Therefore, we give a plan to carry out such changes in small, incremental steps, in order not to affect existing features. We want to apply it to batch workload first. The details can be found in our proposal .
For the past months, we have developed an initial implementation of the above ideas. Initial performance evaluations on TPC-H benchmarks show that substantial performance improvements can be obtained by vectorization (see the figure below). More details can be found in our proposal.
Special thanks to @Kurt Young’s team for all the kind help.
Special thanks to @Piotr Nowojski for all the valuable feedback and help suggestions.
