[SPARK-35266] Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Tests
• Labels: easyfix
• Assignee: Byungsoo Oh
• Reporter: Byungsoo Oh
• Created: 29/Apr/21 04:32
• Updated: 12/Dec/22 18:10
• Resolved: 03/May/21 09:06

Description

When submitting a benchmark job using org.apache.spark.benchmark.Benchmarks class with SPARK_GENERATE_BENCHMARK_FILES=1 option, an exception is raised if the directory where the benchmark file will be generated does not exist.
For example, if you execute BLASBenchmark like the command below, you get an error unless you manually create benchmarks/ directory under spark/mllib-local/.

SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit \
--driver-memory 6g --class org.apache.spark.benchmark.Benchmarks \
--jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \
"`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
"org.apache.spark.ml.linalg.BLASBenchmark"

This is caused by the code in BenchmarkBase.scala where an attempt is made to create the benchmark file without validating the path.

Issue Links

• links to: [Github] Pull Request #32394 (obs0811)
• links to: [Github] Pull Request #32394 (obs0811)
• links to: [Github] Pull Request #32432 (byungsoo-oh)
• links to: [Github] Pull Request #32432 (byungsoo-oh)