[SPARK-41377] Fix spark-version-info.properties not found on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core

Environment

Windows 10

Language

XML powershell

• Assignee: Gautham Banasandra
• Reporter: Gautham Banasandra
• Created: 04/Dec/22 15:19
• Updated: 09/Dec/22 10:08
• Resolved: 09/Dec/22 10:07

Description

What is the problem?

The spark-version-info.properties files is read during initialization of SparkContext. The absence of the same causes the Spark AM's launch to fail with the following error -

22/12/04 17:18:24 ERROR ApplicationMaster: User class threw exception: 
java.lang.ExceptionInInitializerError
	at org.apache.spark.package$.<init>(package.scala:93)
	at org.apache.spark.package$.<clinit>(package.scala)
	at org.apache.spark.SparkContext.$anonfun$new$1(SparkContext.scala:195)
	at org.apache.spark.internal.Logging.logInfo(Logging.scala:60)
	at org.apache.spark.internal.Logging.logInfo$(Logging.scala:59)
	at org.apache.spark.SparkContext.logInfo(SparkContext.scala:84)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:195)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2739)
	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:978)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:972)
	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:747)
Caused by: org.apache.spark.SparkException: Could not find spark-version-info.properties
	at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:62)
	at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala)
	... 18 more

Why did the problem happen?

While building Spark, the spark-version-info.properties file is generated using bash. In Windows environment, if Windows Subsystem for Linux (WSL) is installed, it somehow overrides the other bash executables in the PATH, as noted in SPARK-40739. The bash in WSL has a different mounting configuration and thus, the target location specified for spark-version-info.properties won't be the expected location. Ultimately, this leads to spark-version-info.properties to get excluded from the spark-core jar, thus causing the SparkContext initialization to fail with the above depicted error message.

How can the problem be solved?

I propose to fix this issue by enhancing the build system to automatically detect and switch between using Powershell for Windows and Bash for non-Windows OS. This yields a seamless build experience for all the platform users. Particularly for Windows users, to not worry about the conflicts with different bash installations.

Issue Links

• links to: [Github] Pull Request #38903 (GauthamBanasandra)