[SPARK-45600] Make Python data source registration session level

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 19/Oct/23 00:25
• Updated: 26/Dec/23 02:36
• Resolved: 23/Nov/23 02:04

Description

Currently, registered data sources are stored in `sharedState` and can be accessed across multiple sessions. This, however, will not work with Spark Connect. We should make this registration session level, and support static registration (e.g. using pip install) in the future.

Issue Links

• links to: GitHub Pull Request #43742
• links to: GitHub Pull Request #44487