[SPARK-49687] Delay sorting in validateAndMaybeEvolveStateSchema

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.3
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Zhihong Yu
• Reporter: Zhihong Yu
• Created: 17/Sep/24 14:49
• Updated: 17/Sep/24 18:02
• Resolved: 17/Sep/24 17:58

Description

In `validateAndMaybeEvolveStateSchema`, existing schema and new schema are sorted by column family name.
The sorting can be delayed until `createSchemaFile` is called.

When computing `colFamiliesAddedOrRemoved`, we can use `toSet` to compare column families.

Issue Links

• links to: GitHub Pull Request #48116