[SPARK-45996] Show proper dependency requirement messages for Spark Connect

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 20/Nov/23 01:17
• Updated: 20/Nov/23 04:03
• Resolved: 20/Nov/23 04:03

Description

./bin/pyspark --remote local

We should improve the error messages below.

/.../pyspark/shell.py:57: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/.../pyspark/shell.py", line 52, in <module>
    spark = SparkSession.builder.getOrCreate()
  File "/.../pyspark/sql/session.py", line 476, in getOrCreate
    from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
  File "/.../pyspark/sql/connect/session.py", line 53, in <module>
    from pyspark.sql.connect.client import SparkConnectClient, ChannelBuilder
  File "/.../pyspark/sql/connect/client/__init__.py", line 22, in <module>
    from pyspark.sql.connect.client.core import *  # noqa: F401,F403
  File "/.../pyspark/sql/connect/client/core.py", line 51, in <module>
    import google.protobuf.message
ModuleNotFoundError: No module named 'google

/.../pyspark/shell.py:57: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/.../pyspark/shell.py", line 52, in <module>
    spark = SparkSession.builder.getOrCreate()
  File "/.../pyspark/sql/session.py", line 476, in getOrCreate
    from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
  File "/.../pyspark/sql/connect/session.py", line 53, in <module>
    from pyspark.sql.connect.client import SparkConnectClient, ChannelBuilder
  File "/.../pyspark/sql/connect/client/__init__.py", line 22, in <module>
    from pyspark.sql.connect.client.core import *  # noqa: F401,F403
  File "/.../pyspark/sql/connect/client/core.py", line 52, in <module>
    from grpc_status import rpc_status
ModuleNotFoundError: No module named 'grpc_status'

Issue Links

• links to: GitHub Pull Request #43894