[FLINK-36184] Transform Operator swallows schema changes from tables not present in transform rules

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: cdc-3.2.0, cdc-3.3.0
• Fix Version/s: cdc-3.2.0
• Component/s: Flink CDC
• Labels: pull-request-available
• Assignee: yux
• Reporter: yux
• Created: 30/Aug/24 01:59
• Updated: 30/Aug/24 03:44
• Resolved: 30/Aug/24 03:44

Description

Currently, tables that are not present in transform blocks should be treated as if there's such a dummy fallback block:

transform:

• source-table: "\..\." # capture all unmentioned tables
  projection: "*" # keep all columns

1. without filtering any rows
   There's a bug in #3557's implementation, where schema change events should be filtered out if there's no wildcard in transform rules. However, it also filters out those tables that are not defined in transform rules, which causes schema change events lost with the following example:

transform:

• source-table: foo.bar.baz # Another table that doesn't really matter
  projection: ...
  Here, since there's one transform block, TransformOperator will be added into operator chain. Now let's perform some schema change events in another table (like db.table), it will be filtered out since TransformOperator regards it as a asterisk-less table and does not require schema change events.

By checking if a table is transformed or not, we could set hasAsterisk flag map correctly and resolve this problem.

Issue Links

• links to: GitHub Pull Request #3589