[FLINK-21833] TemporalRowTimeJoinOperator.java will lead to the state expansion by short-life-cycle & huge RowData, although config idle.state.retention.time

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.2
• Fix Version/s: 1.13.0, 1.12.3
• Component/s: Table SQL / Runtime
• Labels: pull-request-available
• Assignee: Lyn Zhang
• Reporter: Lyn Zhang
• Created: 17/Mar/21 03:07
• Updated: 19/Jan/22 02:55
• Resolved: 08/Apr/21 14:48

Description

In our company cases, short-life-cycle & huge RowData will use this operator to join each other. Every key called session_id will be expired after 2min.

With idle.state.retention.time configuration, after 2min if not used, the leftState and rightState will be cleaned up by the operator but nextLeftIndex & registeredTimer state data will be stored forever.

After running a day(about 20 million session_id in our company cases), the checkpoint operator will cause the job crash.

I have found the bug, and fixed it.

Issue Links

• Blocked: FLINK-25695 TemporalJoin cause state leak in some cases
• links to: GitHub Pull Request #15247