[SPARK-48073] StateStore schema incompatibility between 3.2 and 3.4

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.4
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 01/May/24 16:18
• Updated: 05/May/24 20:01

Description

One our customer encountered some schema incompatibility problems when upgrading from Spark 3.2 to 3.4 with structured streaming application.

It seems in 3.4 `Encoders.bean()` includes properties with only getter with or without setter, whereas in 3.2, only properties with both getter and setter are included.

For example, here are schemas for an AtomicLong property/field generated by each version:
3.2: StructType(StructField(opaque,LongType,true),StructField(plain,LongType,true))
3.4: StructType(StructField(acquire,LongType,false),StructField(andDecrement,LongType,false),StructField(andIncrement,LongType,false),StructField(opaque,LongType,false),StructField(plain,LongType,false))

Note that the null ability flag also changes.
Primitive long schema has nullable=true in 3.2, but false in 3.4.

I am not sure if the issue is aware by the community before, and if there is workaround for that?

Thanks.