Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18665

Filesystem connector should use TableSchema exclude computed columns

    XMLWordPrintableJSON

    Details

      Description

      This is reported in http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-use-a-nested-column-for-CREATE-TABLE-PARTITIONED-BY-td36796.html

      create table navi (
        a STRING,
        location ROW<lastUpdateTime BIGINT, transId STRING>
      ) with (
        'connector' = 'filesystem',
        'path' = 'east-out',
        'format' = 'json'
      )
      CREATE TABLE output (
        `partition` AS location.transId
      ) PARTITIONED BY (`partition`)
      WITH (
        'connector' = 'filesystem',
        'path' = 'east-out',
        'format' = 'json'
      ) LIKE navi (EXCLUDING ALL)
      tEnv.sqlQuery("SELECT type, location FROM navi").executeInsert("output")
      

      It throws the following exception

      Exception in thread "main" org.apache.flink.table.api.ValidationException: The field count of logical schema of the table does not match with the field count of physical schema
      . The logical schema: [STRING,ROW<`lastUpdateTime` BIGINT, `transId` STRING>]
      The physical schema: [STRING,ROW<`lastUpdateTime` BIGINT, `transId` STRING>,STRING].
      

      The reason is that FileSystemTableFactory#createTableSource should use schema excluded computed column, not the original catalog table schema.

      [1]: https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemTableFactory.java#L78

        Attachments

          Activity

            People

            • Assignee:
              Leonard Xu Leonard Xu
              Reporter:
              jark Jark Wu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: