[FLINK-7755] Null values are not correctly handled by batch inner and outer joins

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.2, 1.4.0
• Fix Version/s: 1.4.0
• Component/s: Table SQL / API
• Assignee: Fabian Hueske
• Reporter: Fabian Hueske
• Created: 02/Oct/17 19:30
• Updated: 24/Oct/17 18:34
• Resolved: 24/Oct/17 18:34

Description

Join predicates of batch joins are not correctly evaluated according to three-value logic.
This affects inner as well as outer joins.

The problem is that some equality predicates are only evaluated by the internal join algorithms of Flink which are based on TypeComparator. The field TypeComparator for Row are implemented such that null == null results in TRUE to ensure correct ordering and grouping. However, three-value logic requires that null == null results to UNKNOWN (or null). The code generator implements this logic correctly, but for equality predicates, no code is generated.

For outer joins, the problem is a bit tricker because these do not support code-generated predicates yet (see FLINK-5520). FLINK-5498 proposes a solution for this issue.

We also need to extend several of the existing tests and add null values to ensure that the join logic is correctly implemented.

Issue Links

• links to: GitHub Pull Request #4858