[SPARK-38412] `from` and `to` is swapped in the StateSchemaCompatibilityChecker

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2, 3.2.1, 3.3.0
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 04/Mar/22 06:06
• Updated: 09/Mar/22 02:38
• Resolved: 09/Mar/22 02:38

Description

During the work in SPARK-38204, I figured out the parameter is swapped which led to test failure on new test (I disabled the schema check for now in the PR of SPARK-38204).

That allows nullable column to be stored into non-nullable column, which should be prohibited. This is less likely making runtime problem since state schema is conceptual one and row can be stored even not respecting the state schema.

The worse problem is happening in opposite way, that disallows non-nullable column to be stored into nullable column, which should be allowed. Spark fails the query for this case.

We should fix this to allow the case properly.

Issue Links

• links to: [Github] Pull Request #35731 (HeartSaVioR)