[FLINK-36671] Kinesis Table API for new KDS source does not support metadata.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: aws-connector-5.0.0
• Component/s: Connectors / Kinesis
• Assignee: Unassigned
• Reporter: Hong Liang Teoh
• Created: 07/Nov/24 13:28
• Updated: 07/Nov/24 13:29

Description

What

The old Kinesis Table API source `kinesis-legacy` supports defining virtual columns using table metadata.

For example:

 

CREATE TABLE KinesisTable (
  `user_id` BIGINT,
  `arrival_time` TIMESTAMP(3) METADATA FROM 'timestamp' VIRTUAL,
  `shard_id` VARCHAR(128) NOT NULL METADATA FROM 'shard-id' VIRTUAL,
  `sequence_number` VARCHAR(128) NOT NULL METADATA FROM 'sequence-number' VIRTUAL
)
WITH (
  'connector' = 'kinesis-legacy',
  ...
)

We should support this for the new kinesis connector source too.

 

Success Criteria

1. kinesis Table API source supports VIRTUAL columns
2. Automated tests added to validate that the connector supports metadata.
3. Update Table API docs to indicate the bug has been fixed.