[ARROW-13606] [R] Actually disable LTO

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 6.0.0, 5.0.1
• Component/s: R
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/29248
• Assignee: Neal Richardson
• Reporter: Neal Richardson
• Created: 11/Aug/21 13:02
• Updated: 11/Jan/23 08:34
• Resolved: 11/Aug/21 21:05

Description

Background: ARROW-12853 reported that if the C++ code in the R package is built with -flto, it compiles and links successfully, but then the package segfaults on load. ARROW-13199 attempted to fix this by adding -fno-lto to the build, and it did work for the ubuntu 21.04 CI job. However, the LTO check on CRAN failed.

The reason for the difference between the passing ubuntu 21.04 CI job and the CRAN check turns out to be that ARROW-13199 only actually added -fno-lto to the link step. It intended to add them to the CXXFLAGS for compiling too, but it set the wrong variable, so they were not passed in. Yet, the build was still successful because the CXXFLAGS used on the ubuntu 21.04 build included -ffat-lto-objects in addition to -flto. fat-lto-objects can be linked with or without LTO, so the -fno-lto at link time was able to work. The CRAN machine configuration does not have -ffat-lto-objects as a compile flag, so the -fno-lto in the linker meant that linking would fail.

There are two solutions: fix the segfault-on-load and allow LTO or properly disable LTO. This issue is to do the latter; we are continuing to work on the former in a separate issue. ARROW-13507 tried to fix this by setting UseLTO: false in DESCRIPTION, but it turns out that the CRAN check overrides that flag and forces LTO anyway, so that isn't a viable option. And we have to add -ffat-lto-objects instead of -fno-lto because R CMD INSTALL --use-LTO appends -flto at the end, which seems to override -fno-lto if we set it in the package Makevars. But -ffat-lto-objects complements -flto and lets the package link with or without LTO.

Example output from the ubuntu-21.04 build (compiles with -flto=auto -ffat-lto-objects, links with -flto=10 -fno-lto, result is successful non-LTO library

...
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I/arrow/r/check/arrow.Rcheck/00_pkg_src/arrow/libarrow/arrow-5.0.0.9000/include  -Werror -DARROW_R_WITH_ARROW -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -I'/usr/local/lib/R/site-library/cpp11/include'    -fpic  -g -O2 -ffile-prefix-map=/build/r-base-aXXzqd/r-base-4.1.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c type_infer.cpp -o type_infer.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -Wl,-z,relro -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/arrow/r/check/arrow.Rcheck/00_pkg_src/arrow/libarrow/arrow-5.0.0.9000/lib -larrow_dataset -lparquet -larrow -larrow -larrow_bundled_dependencies -larrow_dataset -lparquet -fno-lto -L/usr/lib/R/lib -lR
...

Example output from the CRAN check (compiles with -flto=10, links with -fno-lto, so it fails:

...
g++ -std=gnu++11 -I"/data/gannet/ripley/R/R-devel/include" -DNDEBUG -I/data/gannet/ripley/R/packages/tests-LTO/arrow/libarrow/arrow-5.0.0/include  -DARROW_R_WITH_ARROW -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -I'/data/gannet/ripley/R/test-4.2/cpp11/include' -I/usr/local/include   -fpic  -g -O2 -Wall -pedantic -mtune=native -Wno-ignored-attributes -Wno-parentheses -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong -fstack-clash-protection -fcf-protection -flto=10 -c type_infer.cpp -o type_infer.o
g++ -std=gnu++11 -shared -L/usr/local/lib64 -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o buffer.o chunkedarray.o compression.o compute.o config.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o py-to-r.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/data/gannet/ripley/R/packages/tests-LTO/arrow/libarrow/arrow-5.0.0/lib -larrow_dataset -lparquet -larrow -larrow -larrow_bundled_dependencies -larrow_dataset -lparquet -fno-lto
/usr/bin/ld: RTasks.o: plugin needed to handle lto object
/usr/bin/ld: RTasks.o: plugin needed to handle lto object
/usr/bin/ld: altrep.o: plugin needed to handle lto object
...

Issue Links

• is superceded by: ARROW-13608 [R] symbol initialization appears to be depending on undefined behavior
• links to: GitHub Pull Request #10908
• links to: GitHub Pull Request #10918