Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
table-store-0.3.0
Description
When using flink to query the fts table, we can config the scan parallelism by set the scan.parallelism, but the user may do not know how much parallelism should be used, setting a too large parallelism will cause resource waste, setting the parallelism too small will cause the query to be slow, so we can add parallelism infer.
The function is enabled by default. the parallelism is equal to the number of read splits. Of course, the user can manually turn off the infer function. In order to prevent too many datafiles from causing excessive parallelism, we also set a max infer parallelism. When the infer parallelism exceeds the setting, use the max parallelism.
In addition, we also need to compare with the limit in the select query statement to get a more appropriate parallelism in the case of limit pushdown, for example we have a sql select * from table limit 1, and finally we infer the parallelism is 10, but we only one parallel is needed , besause we only need one data .