[SPARK-38206] Relax the requirement of data type comparison for keys in stream-stream join

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2, 3.2.1, 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 14/Feb/22 11:22
• Updated: 22/Feb/22 07:00
• Resolved: 22/Feb/22 07:00

Description

Currently, stream-stream join checks for the data type compatible between left keys and right keys. It is done as "strict" checking, requiring nullability as same for both sides. This leads to throw assertion error if optimizer turns some columns in one side from nullable to non-nullable but not touching opposite side.

If it is logically correct to relax the nullability check (with deciding proper type on output schema), we should do it to avoid any possible issue from optimization.

Issue Links

• links to: [Github] Pull Request #35599 (HeartSaVioR)