[SPARK-27544] Fix Python test script to work on Scala-2.12 build

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1, 2.4.2
• Fix Version/s: 2.4.3
• Component/s: PySpark, Tests
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 23/Apr/19 06:14
• Updated: 23/Apr/19 22:37
• Resolved: 23/Apr/19 14:01

Description

Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. This issue aims to fix the Python test script on Scala-2.12 build.

$ dev/change-scala-version.sh 2.12

$ build/sbt -Pscala-2.12 package

$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Traceback (most recent call last):
File "python/run-tests.py", line 69, in <module>
raise Exception("Cannot find assembly build directory, please build Spark first.")
Exception: Cannot find assembly build directory, please build Spark first.

Issue Links

• relates to: SPARK-27274 Refer to Scala 2.12 in docs; deprecate Scala 2.11 support in 2.4.1
• links to: GitHub Pull Request #24439