[FLINK-20105] Temporal Table does not work when Kafka is used as the versioned side (planner PK problem)

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.12.0
• Component/s: Connectors / Kafka, Table SQL / Planner
• Assignee: Leonard Xu
• Reporter: Benoît Paris
• Created: 12/Nov/20 10:27
• Updated: 09/Apr/21 02:33
• Resolved: 09/Apr/21 02:33

Description

 

This is probably an early bug report, that I'm making before 0.12 is out. 

In 1.12-SNAPSHOT, doing this:

 

INSERT INTO output_table
SELECT
 o.f_sequence * r.f_int,
 o.f_random_str
FROM datagen_1 AS o
LEFT JOIN input_2 FOR SYSTEM_TIME AS OF o.ts r
 ON o.f_random = r.f_int_pk

 

works when input_2 is build with datagen, but fails when data comes from kafka; yielding the following error that comes from planner code:

 

Type INT NOT NULL of table field 'f_int_pk' does not match with the physical type INT of the 'f_int_pk' field of the TableSource return type.

 

Included is code for a complete reproduction, with:

• docker-compose file for ZooKeeper and Kafka (latest)
• pom.xml
• OK_TempTableSQLTestDatagen.java: it works with the datagen
• KO_TempTableSQLTestKafka.java: fails with Kafka
• KO_TempTableSQLTestKafkaNoPK.java: I tried to have no PK, it fails
• KO_TempTableSQLTestKafkaNull.java: I tried with the PK being nullable, it fails
• KO_TempTableSQLTestKafkaNullif.java: I tried with using the PK in a NULLIF(pk, '') as advertised here, but if fails (loses PK powers)

I just can't think of a workaround. I even tried to GROUP BY on the PK.

From memory, the Temporal Table Function suffers from a similar problem; My usual workaround being to do a COALESCE(problematic_temp_table_function_primary_key, null), but it fails here as well (interestingly, it does not fail because of losing PK powers, but because of the NOT NULL planner difference).

It seems like the same problem of having transformations of the same field being NULL and NOT NULL between planner transformations.

---

 

This isssue is probably related to the ongoing FLIP-132 Temporal Table DDL and Temporal Table Join developments.