[FLINK-35256] Pipeline transform ignores column type nullability

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: cdc-3.1.0
• Component/s: Flink CDC
• Labels: pull-request-available
• Assignee: yux
• Reporter: yux
• Created: 28/Apr/24 06:35
• Updated: 29/Apr/24 02:23
• Resolved: 29/Apr/24 02:23

Description

Flink CDC 3.1.0 brought transform feature, allowing column type / value transformation prior to data routing process. However after the transformation, column type marked as `NOT NULL` lost their annotation, causing some downstream sinks to fail since they require primary key to be NOT NULL.

Here's the minimum reproducible example about this problem:

```yaml
source:
type: mysql
...

sink:
type: starrocks
name: StarRocks Sink
...

pipeline:
name: Sync MySQL Database to StarRocks
parallelism: 4

transform:

• source-table: reicigo.\.*
  projection: ID, UPPER(ID) AS UPID

```

In the MySQL source table, primary key column `ID` is marked as `NOT NULL`, but such information was lost at downstream, causing the following exception (see attachment).

Issue Links

• links to: GitHub Pull Request #3272