[CALCITE-4437] The Sort rel should be decorrelated even though it has fetch or limit when it is not inside a Correlate

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.26.0
• Fix Version/s: 1.27.0
• Component/s: core
• Labels: pull-request-available
• Assignee: Thomas Rebele
• Reporter: Ruben Q L
• Created: 11/Dec/20 19:20
• Updated: 03/Jun/21 22:32
• Resolved: 25/Feb/21 08:29

Description

The fix applied for CALCITE-4206 was "too drastic" and it resulted in Sort with fetch/offset being impossible to decorrelate in all cases.
CALCITE-4333 addressed this issue but only partially (when the Sort with fetch/offset is on top on the plan). However, this solution is insufficient, because any Sort with fetch/offset that is not inside a Correlate can be decorrelated.

Check this test in SqlToRelConverterTest (same test as CALCITE-4333, just with an extra LogicalProject on top of the LogicalSort):

  @Test void testProjectSortLimitWithCorrelateInput() {
    final String sql = ""
        + "SELECT ename||deptno FROM\n"
        + "    (SELECT deptno, ename\n"
        + "    FROM\n"
        + "        (SELECT DISTINCT deptno FROM emp) t1,\n"
        + "          LATERAL (\n"
        + "            SELECT ename, sal\n"
        + "            FROM emp\n"
        + "            WHERE deptno = t1.deptno)\n"
        + "    ORDER BY ename DESC\n"
        + "    LIMIT 3)";
    sql(sql).ok();
  }

The current plan is:

LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
  LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
    LogicalProject(DEPTNO=[$0], ENAME=[$1])
      LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{0}])
        LogicalAggregate(group=[{0}])
          LogicalProject(DEPTNO=[$7])
            LogicalTableScan(table=[[CATALOG, SALES, EMP]])
        LogicalProject(ENAME=[$1], SAL=[$5])
          LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
            LogicalTableScan(table=[[CATALOG, SALES, EMP]])

It can actually be decorrelated as:

LogicalProject(EXPR$0=[||($1, CAST($0):VARCHAR NOT NULL)])
  LogicalSort(sort0=[$1], dir0=[DESC], fetch=[3])
    LogicalProject(DEPTNO=[$0], ENAME=[$1])
      LogicalJoin(condition=[=($0, $3)], joinType=[inner])
        LogicalAggregate(group=[{0}])
          LogicalProject(DEPTNO=[$7])
            LogicalTableScan(table=[[CATALOG, SALES, EMP]])
        LogicalProject(ENAME=[$1], SAL=[$5], DEPTNO=[$7])
          LogicalTableScan(table=[[CATALOG, SALES, EMP]])

Issue Links

• is caused by: CALCITE-4206 RelDecorrelator outputs wrong plan for correlate sort with fetch limit
• is related to: CALCITE-4333 The Sort rel should be decorrelated even though it has fetch or limit when its parent is not a Correlate
• links to: GitHub Pull Request #2302