Details
-
Task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
There was an intermittent crash in the deep_learning module due to both tensorflow and madlib using the same STL class (std::set<string>), and the functions in this class being publicly exported from libmadlib.so
We fixed the crash by adding the symbols for the std::set<string> functions to the list of symbols the linker should hide during the build process. But we left a lot of other visible global symbols exported that shouldn't be exported... in order to be avoid extensive testing to make sure it doesn't break anything on an obscure platform or for some configurations.
Without doing this, we will always run the risk that there is another library we are using (could be very indirect, such as with this case where python imports a module which then calls a C library) that conflicts with the version of STL or Boost that madlib is using. If that happens, we could easily have another unexplained crash and it would be very difficult again to track down.
The recommended best practice for exporting symbols from a shared object file is that we should hide everything by default, and only expose those functions which are officially a part of our API.
We can hide everything by default by adding fvisibility=hidden to the list of default params to pass to the compiler in the master CMakeLists.txt. This will cause gcc to hide all symbols from external libraries by default. In addition, we should use either a #pragma or by adding a macro to prefix function definitions with that sets the visibility on specific functions we want to expose to default (visible). We can remove the -unexported_symbols_list property from OSX and -version-script property from other platforms, as these will no longer be needed. We should also delete the files they reference, library.ver and unexported_symbols_list.txt
Two good references on how to control visibility of symbols:
https://developer.apple.com/library/archive/technotes/tn2185/_index.html
https://gcc.gnu.org/wiki/Visibility
Hopefully the list of symbols we need to expose is just the list of sql functions implemented in C, but there may be a few others that also need to be exposed.