[SPARK-44839] Better error logging when user accesses spark session in foreachBatch and Listener

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0, 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, Structured Streaming
• Assignee: Wei Liu
• Reporter: Wei Liu
• Created: 16/Aug/23 23:45
• Updated: 25/Aug/23 09:04
• Resolved: 25/Aug/23 09:04

Description

TypeError: cannot pickle '_thread._local' object

 

when user access `spark`. we need a better error for this