[ARROW-17902] [C++][Gandiva] Solve LLVM 15 CI performace and compatibility issues

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 9.0.0
• Component/s: C++ - Gandiva
• External issue URL: https://github.com/apache/arrow/issues/20439
• Assignee: Unassigned
• Reporter: Jin Shang
• Created: 30/Sep/22 02:46
• Updated: 11/Jan/23 11:56

Description

Package managers have been upgrading their default LLVM version to 15. This raises several compatibility and performance issues for CI, and we currently are using workaounds

1. Mac's default clang 13's LTO cannot read Brew installed llvm static libs because they are compiled with llvm 15 (https://github.com/apache/arrow/pull/14187#issuecomment-1253322948). Workaround: Use brew installed clang: ARROW-17805 ARROW-17842
2. Conda's packaging of LLVM 15 on Windows is not portable. It required a VS2019 Enterprise version: https://github.com/conda-forge/llvmdev-feedstock/issues/175. Workaround: Pin LLVM to 14 for Conda with Windows. ARROW-17830
3. Brew's LLVM15 enables z3 by default and it's shared linked. It breaks Java jar builds on MacOS: https://github.com/apache/arrow/pull/14261#issuecomment-1263024112. Workaround: Pin LLVM to 14 for all MacOS tasks.
4. LLVM 15 is noticably slower to build than LLVM 14 for brew and MacOS. Workaround: Pin LLVM to 14 for all MacOS tasks. ARROW-17872

We need to find a way to resolve these workarounds for LLVM 15.

Edit: The current mac os runner we use `macos-latest` has llvm@14 preinstalled. So pinning to 14 can save build time and fix dependency issues. We can move to llvm@15 on mac once the github images support them by default.

Issue Links

• relates to: ARROW-17805 [C++][CI] Use Brew installed clang for MacOS
• relates to: ARROW-17830 [C++][Gandiva] AppVeyor Windows builds failing due to 'diaguids.lib'
• relates to: ARROW-17842 [C++][CI] Use Brew installed clang for MacOS verify-rc
• relates to: ARROW-17869 [Java][Gandiva] ProjectorTest.testStringOutput is failed
• relates to: ARROW-17872 [CI] Reduce required dependencies on macOS builds