[SPARK-47708] Do not log the gRPC exception to stderr in PySpark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Assignee: Nemanja Boric
• Reporter: Nemanja Boric
• Created: 03/Apr/24 08:18
• Updated: 04/Apr/24 13:48
• Resolved: 04/Apr/24 01:28

Description

Currently if there's any gRPC exception, instead of just handling it, the PySpark's gRPC error handler is going to print it out to the stderr, not allowing the user to cleanly ignore the exception by using try/except control flow statement.

 

Issue Links

• is a clone of: SPARK-47380 Validate the spark connect session on server side
• links to: GitHub Pull Request #45840
• links to: GitHub Pull Request #45878