Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35266

Fix an error in BenchmarkBase.scala that occurs when creating a benchmark file in a non-existent directory

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: Tests
    • Labels:

      Description

      When submitting a benchmark job using org.apache.spark.benchmark.Benchmarks class with SPARK_GENERATE_BENCHMARK_FILES=1 option, an exception is raised if the directory where the benchmark file will be generated does not exist.
      For example, if you execute BLASBenchmark like the command below, you get an error unless you manually create benchmarks/ directory under spark/mllib-local/.

      SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit \
      --driver-memory 6g --class org.apache.spark.benchmark.Benchmarks \
      --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \
      "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
      "org.apache.spark.ml.linalg.BLASBenchmark"
      

      This is caused by the code in BenchmarkBase.scala where an attempt is made to create the benchmark file without validating the path.

        Attachments

          Activity

            People

            • Assignee:
              byungsoo.oh Byungsoo Oh
              Reporter:
              byungsoo.oh Byungsoo Oh
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: