[CALCITE-3071] Cache the whole sql plan to reduce the latency and improve the performance

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.19.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Lai Zhou
• Created: 16/May/19 02:54
• Updated: 25/Aug/21 20:41

Description

In real business, when sql queries become complex,  the overhead of sql plan will increase quickly , and many of the sql queries are duplicates.

We already have some caching  issue about improving the performance, such as the issue

https://issues.apache.org/jira/browse/CALCITE-2703,

which reduce code generation and class loading overhead when executing queries in the EnumerableConvention, but I think it's not enough.

I propose to cache the whole sql plan to reduce the latency ，for the same sql

, ignoring the cost optimizing based on statistics here, we can cache the generated code for it.

I use the FrameworkConfig API to execute sql queries, in this way I can easily do this job .

but it's not easy to make a whole sql execution plan(that says code-gen) cache in the sql processing flow based on JDBC Connection, because there're many intermediate state in this processing flow.

 

Let's discuss this feature and the probable solutions.