[SPARK-38522] Strengthen the contract on iterator method in StateStore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 11/Mar/22 07:48
• Updated: 23/Mar/22 01:09
• Resolved: 23/Mar/22 01:09

Description

The root cause of SPARK-38320 was that the logic initialized the iterator first, and performed some updates against state store, and iterated through iterator expecting that all updates in between should be visible in iterator.

That is not guaranteed in RocksDB state store, and the contract of Java ConcurrentHashMap which is used in HDFSBackedStateStore does not also guarantee it.

It would be clearer if we update the contract to draw a line on behavioral guarantee to callers so that callers don't get such expectation.

Issue Links

• links to: [Github] Pull Request #35816 (HeartSaVioR)
• links to: [Github] Pull Request #35816 (HeartSaVioR)