[FLINK-33523] DataType ARRAY<INT NOT NULL> fails to cast into Object[]

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.18.0
• Fix Version/s: 1.18.1
• Component/s: Table SQL / API
• Labels: pull-request-available
• Assignee: Martijn Visser
• Reporter: Prabhu Joseph
• Created: 11/Nov/23 03:45
• Updated: 01/Feb/24 09:41
• Resolved: 08/Dec/23 08:18

Description

When upgrading Iceberg's Flink version to 1.18, we found the Flink-related unit test case broken due to this issue. The below code used to work fine in Flink 1.17 but failed after upgrading to 1.18. DataType ARRAY<INT NOT NULL> fails to cast into Object[].

Error:

Exception in thread "main" java.lang.ClassCastException: [I cannot be cast to [Ljava.lang.Object;
at FlinkArrayIntNotNullTest.main(FlinkArrayIntNotNullTest.java:18)

Repro:

  import org.apache.flink.table.data.ArrayData;
  import org.apache.flink.table.data.GenericArrayData;
  import org.apache.flink.table.api.EnvironmentSettings;
  import org.apache.flink.table.api.TableEnvironment;
  import org.apache.flink.table.api.TableResult;

  public class FlinkArrayIntNotNullTest {

    public static void main(String[] args) throws Exception {

      EnvironmentSettings settings = EnvironmentSettings.newInstance().inBatchMode().build();
      TableEnvironment env = TableEnvironment.create(settings);

      env.executeSql("CREATE TABLE filesystemtable2 (id INT, data ARRAY<INT NOT NULL>) WITH ('connector' = 'filesystem', 'path' = '/tmp/FLINK/filesystemtable2', 'format'='json')");
      env.executeSql("INSERT INTO filesystemtable2 VALUES (4,ARRAY [1,2,3])");
      TableResult tableResult = env.executeSql("SELECT * from filesystemtable2");

      ArrayData actualArrayData = new GenericArrayData((Object[]) tableResult.collect().next().getField(1));
    }
  }

Analysis:

1. The code works fine with ARRAY<INT> datatype. The issue happens when using ARRAY<INT NOT NULL>.
2. The code works fine when casting into int[] instead of Object[].

Issue Links

• is caused by: FLINK-31835 DataTypeHint don't support Row<i Array<int>>
• is duplicated by: FLINK-33547 SQL primitive array type after upgrading to Flink 1.18.0
• links to: GitHub Pull Request #23890
• links to: GitHub Pull Request #23891