Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.4.0
-
None
Description
There are a number of issues related to building Kudu on Centos/RHEL 7 with devtoolset, and on Centos/RHEL 7 machines with devtoolset installed (but not enabled).
A couple of bits of background info:
1) RHEL 7 ships with system GCC/libstdcxx version 4.8.
2) devtoolset-3 ships with GCC/libstdcxx version 4.9.
3) devtoolset-6 ships with GCC/libstdcxx version 6.2.
4) GCC incompatibly changed the unordered_set and unordered_map ABIs between libstdcxx version 4.8 and 4.9. In 4.8, sizeof(unordered_set<int>) is 48 bytes, whereas in 4.9 and above it is 56 bytes (and similarly for unordered_map).
5) Clang will automatically search for devtoolset installations, and if found, use those headers as opposed to the system headers.
6) Clang 4.0 (the current version bundled by Kudu) will find devtoolset versions up to devtoolset-4. The next version of Clang (Clang 5) will find devtoolset version up to devtoolset-6.
7) G++ compilers provided by a devtoolset will use the libstdcxx headers provided by that devtoolset.
8) Among other things, Kudu uses its bundled clang to pre-compile C++ source files into clang bitcode. One of the classes included in this pre-compilation is kudu::Schema, which includes an unordered_set field.
As a result, with certain configurations, the precompiled codegen will cause crashes at runtime:
Kudu compiled with system gcc on a Centos 7 machine with devtoolset-3 installed: Kudu will be compiled against the system headers, where unordered_set is 48 bytes. The precompiled code will be compiled by clang against the devtoolset-3 headers, where unordered_set is 56 bytes. This results in runtime crashes when calling in to codegenerated functions (codegen-test segfaults reliably).
Kudu compiled with devtoolset-6 gcc on Centos7: Kudu will be compiled against the devtoolset-6 headers, where unordered_set is 54 bytes. The precompiled code will be compiled by clang against the system headers, where unordered_set is 48 bytes (clang 4.0 will not find the devtoolset-6 installation). This results in runtime crashes when calling in to code generated functions (codegen-test segfaults reliably).
Passing the -gcc-toolchain flag to clang with a value appropriate for the currently enabled g++ when precompiling sources will fix this issue, but I haven't found a clean way to figure out how to determine the 'correct' value as part of a script. For system gcc the value should be /usr, and for devtoolset-6 the value should be /opt/rh/devtoolset-6/root/usr. This corresponds to the --prefix flag value under the configure flags that gcc -v spits out, so maybe we can parse it from that.
Another option is to replace unordered_set/unordered_map fields in any objects passed through codegen boundaries, but obviously this is brittle (and there may be other types out there whose ABI is similarly unstable).
Finally, I'll note that this issue is a good reason why we should maintain the no-c++11 types in public-APIs rule that we currently follow with the client. Also related: devtoolset ABI guarantees.
Attachments
Issue Links
- relates to
-
-
- Open
-