[FLINK-35318] incorrect timezone handling for TIMESTAMP_WITH_LOCAL_TIME_ZONE type during predicate pushdown

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.18.1
• Component/s: Table SQL / API
• Labels: pull-request-available

Environment

flink version 1.18.1

iceberg version 1.15.1

• Assignee: linshangquan
• Reporter: linshangquan
• Created: 09/May/24 06:18
• Updated: 21/May/24 06:45

Description

In our scenario, we have an Iceberg table that contains a column named 'time' of the timestamptz data type. This column has 10 rows of data where the 'time' value is '2024-04-30 07:00:00' expressed in the "Asia/Shanghai" timezone.

 

We encountered a strange phenomenon when accessing the table using Iceberg-flink.

When the WHERE clause includes the time column, the results are incorrect.

ZoneId.systemDefault() = "Asia/Shanghai" 

When there is no WHERE clause, the results are correct.

During debugging, we found that when a WHERE clause is present, a FilterPushDownSpec is generated, and this FilterPushDownSpec utilizes RexNodeToExpressionConverter for translation.

When RexNodeToExpressionConverter#visitLiteral encounters a TIMESTAMP_WITH_LOCAL_TIME_ZONE type, it uses the specified timezone "Asia/Shanghai" to convert the TimestampString type to an Instant type. However, the upstream TimestampString data has already been processed in UTC timezone. By applying the local timezone processing here, an error occurs due to the mismatch in timezones.

Whether the handling of TIMESTAMP_WITH_LOCAL_TIME_ZONE type of data in RexNodeToExpressionConverter#visitLiteral is a bug, and whether it should process the data in UTC timezone.

 
Please help confirm if this is the issue, and if so, we can submit a patch to fix it.
 
 

Issue Links

• links to: GitHub Pull Request #24787