Currently benchmark can be done in two ways: spark-submit, or SBT command. However in the former Spark will miss some properties such as IS_TESTING, which is useful to turn on/off some behavior like codegen. Therefore, the result could differ with the two methods. In addition, the benchmark GitHub workflow is using the spark-submit approach.
This propose to set IS_TESTING to true in BenchmarkBase so that it is always on.