[SPARK-47311] Suppress Python exceptions where PySpark is not in the Python path

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SQL
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 07/Mar/24 01:28
• Updated: 07/Mar/24 20:33
• Resolved: 07/Mar/24 04:09

Description

scala> sql("create table t(i int) using json")
24/03/06 16:09:44 WARN DataSourceManager: Skipping the lookup of Python Data Sources due to the failure.
org.apache.spark.SparkException:
Error from python worker:
  /opt/homebrew/Caskroom/miniconda/base/bin/python3: Error while finding module specification for 'pyspark.daemon' (ModuleNotFoundError: No module named 'pyspark')

When PySpark is not in the Python path at all, it always shows this warning message once for every Spark session initialization.

Issue Links

• links to: GitHub Pull Request #45414