[SPARK-31990] Streaming's state store compatibility is broken

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: Structured Streaming
• Labels: correctness
• Target Version/s: 3.0.1
• Assignee: Takeshi Yamamuro
• Reporter: Xiao Li
• Created: 14/Jun/20 19:10
• Updated: 15/Jun/20 14:49
• Resolved: 15/Jun/20 14:49

Description

This line of https://github.com/apache/spark/pull/28062 changed the order of groupCols in dropDuplicates(). Thus, the executor JVM could probably crash, throw a random exception or even return a wrong answer when using the checkpoint written by the previous version. 

Issue Links

• blocks: SPARK-31894 Introduce UnsafeRow format validation for streaming state store
• incorporates: SPARK-31905 Add compatibility tests for streaming state store format
• is caused by: SPARK-31292 Replace toSet.toSeq with distinct for readability
• relates to: SPARK-19497 dropDuplicates with watermark
• links to: [Github] Pull Request #28830 (maropu)
• links to: [Github] Pull Request #28830 (maropu)