[SPARK-25651] spark-shell gets wrong version of spark on windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.3.2
• Component/s: Spark Shell

Environment

Windows 10, running spark 2.3.2

• Assignee: Unassigned
• Reporter: Nick Sutcliffe
• Created: 05/Oct/18 16:54
• Updated: 12/Dec/22 18:10
• Resolved: 08/Oct/18 05:42

Description

I have multiple versions of spark on my computer, and in particular SPARK_HOME set to a spark 2.0.2 installation.

If I browse to the bin directory of my spark 2.3.2 installation and run spark-shell, it incorrectly uses my spark 2.0.2 installation for SPARK_HOME. It seems that in spark-shell2.cmd, previously it set SPARK_HOME as follows (verified in spark 2.0.2 and spark 2.2.0):

`set SPARK_HOME=%~dp0..`

However this is not present in spark 2.3.2, instead calling find-spark-home.cmd which appears to be incorrectly assuming to take the environment variable if it exists.