Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Implemented
-
Impala 4.2.0
-
None
-
ghx-label-4
Description
It is desirable to be able to iterate quickly by running "make -j impalad" while modifying a file. Currently, modifying most files incurs a rebuild of the LLVM IR, which is a slow serial step. For example:
$ touch be/src/runtime/coordinator.cc $ make -j impalad ... [ 98%] Generating ../../../llvm-ir/impala.bc [ 98%] Generating ../../../llvm-ir/impala-legacy-avx.bc [ 98%] Generating ../../generated-sources/impala-ir/impala-ir.cc [ 98%] Generating ../../generated-sources/impala-ir/impala-ir-legacy-avx.cc ...
This can add several seconds to an incremental build. This step happens for files that do not actually impact the LLVM IR, so there are ways to avoid this.
The reason that LLVM IR is rebuilt is because it has a dependencies on Exec, Exprs, Runtime, Udf, Util, and other libraries:
add_custom_command(
OUTPUT ${IR_OUTPUT_FILE}
COMMAND ${LLVM_CLANG_EXECUTABLE} ${CLANG_IR_CXX_FLAGS} ${PLATFORM_SPECIFIC_FLAGS}
${CLANG_INCLUDE_FLAGS} ${IR_INPUT_FILES} -o ${IR_TMP_OUTPUT_FILE}
COMMAND ${LLVM_OPT_EXECUTABLE} ${LLVM_OPT_IR_FLAGS} < ${IR_TMP_OUTPUT_FILE} > ${IR_OUTPUT_FILE}
COMMAND rm ${IR_TMP_OUTPUT_FILE}
DEPENDS Exec ExecAvro ExecKudu Exprs Runtime Udf Util ${IR_INPUT_FILES}
)
From a correctness perspective, the LLVM IR only cares about things that impact the content of the *-ir.cc files, because impala-ir.cc includes every *-ir.cc file. That list of libraries is a superset of what is needed.
If the *-ir.cc files were split off into their own libraries (i.e. ExecIr rather than Exec), then this target would only depend on the ExecIr rather than the larger Exec. This would reduce the number of files that would cause LLVM IR to be rebuilt. That should reduce the runtime of an incremental "make -j impalad" for quite a few C++ files.