[SPARK-38809] Implement option to skip null values in symmetric hash impl of stream-stream joins

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.0, 3.2.2
• Component/s: Structured Streaming
• Assignee: Anish Shrigondekar
• Reporter: Anish Shrigondekar
• Created: 06/Apr/22 18:38
• Updated: 11/Apr/22 07:28
• Resolved: 08/Apr/22 04:08

Description

Implement option to skip null values in symmetric hash impl of stream-stream joins

• In the symmetric has join state manager, we can receive entries with null values for a key and that caused the `removeByValue` and get iterators to fail and run into the NullPointerException.
• This is possible if the state recovered is written from a old spark version or its corrupted on disk. Since we don't have a utility to query this state, we would like to provide a conf option to skip nulls for the symmetric hash impl in stream stream joins.

 

Issue Links

• links to: [Github] Pull Request #36090 (anishshri-db)
• links to: [Github] Pull Request #36090 (anishshri-db)