Minor Description
SparkSubmitSuite test failures can be painful to debug: currently, failed tests result in messages like
org.scalatest.exceptions.TestFailedException: Process returned with exit code 1. See the log4j logs for more detail.
which require the developer to hunt in test logs in order to identify the root cause of the test failure.
In contrast, HiveSparkSubmitSuite's output packs lots more useful information into its test failure messages:
[info] - temporary Hive UDF: define a UDF and use it *** FAILED *** (4 seconds, 135 milliseconds) [info] spark-submit returned with exit code 101. [info] Command line: '/Users/joshrosen/oss-spark/bin/spark-submit' '--class' 'wrongClassName' '--name' 'TemporaryHiveUDFTest' '--master' 'local-cluster[2,1,1024]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--driver-java-options' '-Dderby.system.durability=test' '--jars' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-32d0a47c-33eb-4488-866a-9994b3727b5b/testJar-1631766233448.jar,file:/Users/joshrosen/oss-spark/target/tmp/spark-e9e32588-83fa-43bd-a60f-9538fd30ab9a/testJar-1631766233601.jar' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-daef4628-d953-41e3-b935-a726a7796418/testJar-1631766232701.jar' 'SparkSubmitClassA' 'SparkSubmitClassB' [info] [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Class path contains multiple SLF4J bindings. [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/oss-spark/assembly/target/scala-2.12/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] [info] 2021-09-15 21:23:54.752 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. [info] 2021-09-15 21:23:54.753 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [info] 2021-09-15 21:23:55.623 - stderr> Error: Failed to load class wrongClassName. (SparkSubmitTestUtils.scala:97) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [...]
This is done via code in SparkSubmitTestUtils, which is currently located in the sql/hive module. I propose a refactoring to move this into the core module so that it can be used by both SparkSubmitSuite and HiveSparkSubmitSuite, enriching the former with nicer error outputs.
