[FLINK-27998] Upgrade Calcite version to 1.30

• Type: Technical Debt
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.18.0
• Component/s: Table SQL / API, Table SQL / Runtime
• Labels: pull-request-available
• Assignee: Sergey Nuyanzin
• Reporter: Martijn Visser
• Created: 10/Jun/22 09:40
• Updated: 19/Apr/23 06:59
• Resolved: 19/Apr/23 06:46

Description

The latest available version of Calcite is currently 1.30. We already need to execute the rework that was planned when upgrading to Calcite 1.27 FLINK-20873 and when upgrading to Calcite 1.28 FLINK-21239

When doing this upgrade, we should do this to the last available version. This is needed to resolve multiple bugs.

Additional note: Flink currently uses Calcite 1.26.0 which has this note on the release page:

Warning: Calcite 1.26.0 has severe issues with RexNode simplification caused by SEARCH operator ( wrong data from query optimization like in CALCITE-4325, CALCITE-4352, NullPointerException), so use 1.26.0 for development only, and beware that Calcite 1.26.0 might corrupt your data.
https://calcite.apache.org/news/2020/10/06/release-1.26.0/

The following files should be removed from the Flink code base when upgrading calcite to 1.30.0

in `org.apahce.calcite.rel.core`:

• Correlate
• Filter
• Intersect
• Minus
• SetOp
• Sort
• Union
• Values
• Window

in `org.apahce.calcite.rel.hint`:

• HintPredicates
• NodeTypeHintPredicate

in `org.apahce.calcite.rel.logical`:

• LogicalCorrelate
• LogicalFilter
• LogicalIntersect
• LogicalMinus
• LogicalSort
• LogicalUnion
• LogicalValues
• LogicalWindow

Issue Links

• causes: FLINK-28518 Exception: AssertionError: Cannot add expression of different type to set in sub-query with ROW type
• is related to: FLINK-31362 Upgrade to Calcite version to 1.33.0
• is related to: FLINK-29319 Upgrade Calcite version to 1.32
• is related to: FLINK-29932 Upgrade Calcite version to 1.29.0
• relates to: FLINK-28678 Support SQL hint for Filter, SetOp, Sort, Window, Values
• relates to: FLINK-20873 Upgrade Calcite version to 1.27
• relates to: FLINK-21239 Upgrade Calcite version to 1.28
• relates to: FLINK-29540 SubQueryAntiJoinTest started to fail after Calcite 1.27
• relates to: FLINK-31349 Adapt to Calcite's 1.30+ new ROW null semantics
• relates to: FLINK-31350 Support Calcite's 1.30+ UnknownType
• relates to: FLINK-28744 Upgrade Calcite version to 1.31
• links to: GitHub Pull Request #21934
• mentioned in: Page Loading...

Sub-Tasks

• Calcite -test.jar are no longer published, use calcite-testkit instead: Resolved
• Remove RexSimplify together with calcite upgrade: Resolved
• Adapt to new calcite rule configuration api: Resolved
• After updating to calcite 1.30.0 and janino 3.1.6 some tests started to fail with compilation issues in generated code: Resolved
• Cast of integer array to string stopped working after upgrade to calcite 1.30: Resolved
• Almost every test inside FlinkSqlParserImplTest is failing after uprgade to calcite of higher versions: Resolved
• NPE after calcite upgrade at org.apache.calcite.tools.RelBuilder.groupKey(RelBuilder.java:1287): Closed
• RexSimplify can not be removed after update to calcite 1.27: Closed