[MESOS-1745] Building Python egg on OS X using clang is problematic

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.19.1

Environment

Mac OS X 10.8.5
clang-3.3 installed via brew
python 2.7.8 install via brew ([GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)])

• Assignee: Unassigned
• Reporter: Jie Yu
• Created: 29/Aug/14 05:20
• Updated: 04/Sep/14 09:09

Description

Here is what I have observed, correct me if I am wrong.

To reproduce:

$ tar zxvf mesos-0.19.1.tar.gz
$ cd mesos-0.19.1
$ mkdir build
$ cd build
$ CC=clang-3.3 CXX=clang++-3.3 ../configure
$ make
$ PYTHONPATH=src/python/dist/mesos-0.19.1-py2.7-macosx-10.8-x86_64.egg python
Python 2.7.8 (default, Aug 18 2014, 15:46:02) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mesos
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.8-x86_64/egg/mesos.py", line 26, in <module>
  File "build/bdist.macosx-10.8-x86_64/egg/_mesos.py", line 7, in <module>
  File "build/bdist.macosx-10.8-x86_64/egg/_mesos.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/jyu/.python-eggs/mesos-0.19.1-py2.7-macosx-10.8-x86_64.egg-tmp/_mesos.so, 2): Symbol not found: __ZN5mesos20MesosSchedulerDriverC1EPNS_9SchedulerERKNS_13FrameworkInfoERKSs
  Referenced from: /Users/jyu/.python-eggs/mesos-0.19.1-py2.7-macosx-10.8-x86_64.egg-tmp/_mesos.so
  Expected in: flat namespace
 in /Users/jyu/.python-eggs/mesos-0.19.1-py2.7-macosx-10.8-x86_64.egg-tmp/_mesos.so
>>>

After digging this problem today, here is what I've found:

1. build/src/.libs/libmesos_no_3rdparty.a is built using clang+-3.3 with flags -std=c11 -stdlib=libc+. So the mangled MesosSchedulerDriver constructor symbol is like the following in libmesos_no_3rdparty.a

[tw-mbp-jyu build]$ nm src/.libs/libmesos_no_3rdparty.a | grep MesosSchedulerDriver
...
00000000000006d0 T __ZN5mesos20MesosSchedulerDriverC1EPNS_9SchedulerERKNS_13FrameworkInfoERKNSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEE
...

2. build/src/python/build/temp.macosx-10.8-x86_64-2.7/native/mesos_scheduler_driver_impl.o is built using clang+-3.3 without -stdlib=libc+

clang-3.3 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -g -Qunused-arguments -Qunused-arguments -I/tmp/build/mesos-0.19.1/build/../include -I/tmp/build/mesos-0.19.1/build/include -I/tmp/build/mesos-0.19.1/build/src -I/tmp/bui
ld/mesos-0.19.1/build/src/python/native -I/tmp/build/mesos-0.19.1/build/3rdparty/libprocess/3rdparty/protobuf-2.5.0/src -I/opt/twitter/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c native/mesos_
scheduler_driver_impl.cpp -o build/temp.macosx-10.8-x86_64-2.7/native/mesos_scheduler_driver_impl.o

As a result, it expects a mangled MesosSchedulerDriver constructor symbol like the following:

[tw-mbp-jyu build]$ nm src/python/build/temp.macosx-10.8-x86_64-2.7/native/mesos_scheduler_driver_impl.o | grep MesosSchedulerDriver
                 U __ZN5mesos20MesosSchedulerDriverC1EPNS_9SchedulerERKNS_13FrameworkInfoERKSs
...

3. As a result, the above symbol will remain 'unresolved' in _mesos.so, and subsequently causes issue when we 'import mesos' using the egg.

Issue Links

• is related to: MESOS-946 setup.py should be able to create publishable sdist
• is related to: AURORA-674 Increment Aurora to depend upon mesos 0.20.0