[SPARK-47788] Ensure the same hash partitioning scheme/hash function is used across batches

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Fanyue Xia
• Reporter: Fanyue Xia
• Created: 09/Apr/24 23:18
• Updated: 30/Apr/24 16:02
• Resolved: 15/Apr/24 06:22

Description

To really make sure that any changes to hash function / partitioner in Spark doesn’t cause logical correctness issues in existing running streaming queries, we should add a new unit test, to ensure hash function stability is maintained.

Issue Links

• links to: GitHub Pull Request #45971
• links to: GitHub Pull Request #46303