[SPARK-41524] Expose SQL confs and extraOptions separately in o.a.s.sql.execution.streaming.state.RocksDBConf

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Huanli Wang
• Reporter: Huanli Wang
• Created: 15/Dec/22 06:03
• Updated: 15/Dec/22 21:06
• Resolved: 15/Dec/22 21:06

Description

Currently the usage of StateStoreConf is via confs, which composes both SQL confs and extraOptions into one. The name of config for extraOptions shouldn't have to follow the name prefix of SQL conf, because it's not bound to the context of SQL conf.

After differentiate SQL conf and extraOptions in StateStoreConf, we should be able to adopt more use case on operator level configs by using the extraOptions.

Issue Links

• links to: [Github] Pull Request #39069 (huanliwang-db)
• links to: [Github] Pull Request #39069 (huanliwang-db)