[FLINK-30282] Logical type ROW lost inner field's nullability after convert to RelDataType

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.16.0, 1.16.1
• Component/s: Table SQL / API
• Labels: pull-request-available
• Assignee: Jane Chan
• Reporter: Jane Chan
• Created: 02/Dec/22 14:29
• Updated: 13/Jun/23 09:43
• Resolved: 09/Jun/23 07:46

Description

Issue History

This is not a new issue, FLINK-13604 has tracked it before, and FLINK-16344 spared efforts to fix it (but did not tweak the ut case mentioned in FLINK-13604, i.e. SqlToOperationConverterTest#testCreateTableWithFullDataTypes). Nevertheless, the FunctionITCase added by FLINK-16344, which validates the fix, has been removed in FLINK-16377.

How to Reproduce

 c.c2 lost nullability

Flink SQL> create table dummy (a array<int not null> not null, b array<string not null>, c row<c1 int, c2 double not null>) with ('connector' = 'datagen');
[INFO] Execute statement succeed.
Flink SQL> desc dummy;
+------+----------------------------+-------+-----+--------+-----------+
| name |                       type |  null | key | extras | watermark |
+------+----------------------------+-------+-----+--------+-----------+
|    a |        ARRAY<INT NOT NULL> | FALSE |     |        |           |
|    b |     ARRAY<STRING NOT NULL> |  TRUE |     |        |           |
|    c | ROW<`c1` INT, `c2` DOUBLE> |  TRUE |     |        |           |
+------+----------------------------+-------+-----+--------+-----------+
3 rows in set

Root Cause

Two places are causing this problem in ExtendedSqlRowTypeNameSpec.

1. dt.deriveType should also pass dt's nullability as well. See https://github.com/apache/flink/blob/fb27e6893506006b9a3b1ac3e9b878fb6cad061a/flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/type/ExtendedSqlRowTypeNameSpec.java#L159
 

2. StructKind should be PEEK_FIELDS_NO_EXPAND instead of FULLY_QUALIFIED(see https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rel/type/StructKind.java), so that FlinkTypeFactory#createTypeWithNullability will not fall back to super implement. See https://github.com/apache/flink/blob/fb27e6893506006b9a3b1ac3e9b878fb6cad061a/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/calcite/FlinkTypeFactory.scala#L417

Issue Links

• is related to: FLINK-31829 Conversion to relational algebra failed to preserve datatypes' nullabilities
• relates to: FLINK-31830 Coalesce on nested fields with different nullabilities will get wrong plan
• links to: GitHub Pull Request #21452