Avro
  1. Avro
  2. AVRO-981

Python Avro library does not build/install on OS X

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.5.4, 1.6.1
    • Fix Version/s: 1.7.3
    • Component/s: python
    • Environment:

      Mac OS X 10.6.8, Python 2.5, 2.6, 2.7

    • Tags:
      python avro mac os x build

      Description

      russell-jurneys-macbook-pro:py rjurney$ sudo python2.5 setup.py install
      Password:
      running install
      running bdist_egg
      running egg_info
      writing requirements to avro.egg-info/requires.txt
      writing avro.egg-info/PKG-INFO
      writing top-level names to avro.egg-info/top_level.txt
      writing dependency_links to avro.egg-info/dependency_links.txt
      reading manifest file 'avro.egg-info/SOURCES.txt'
      writing manifest file 'avro.egg-info/SOURCES.txt'
      installing library code to build/bdist.macosx-10.6-i386/egg
      running install_lib
      running build_py
      creating build/bdist.macosx-10.6-i386
      creating build/bdist.macosx-10.6-i386/egg
      creating build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/_init_.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/datafile.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/io.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/ipc.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/protocol.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/schema.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/tool.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/avro/txipc.py -> build/bdist.macosx-10.6-i386/egg/avro
      copying build/lib/pyAntTasks-1.3-LICENSE.txt -> build/bdist.macosx-10.6-i386/egg
      copying build/lib/pyAntTasks-1.3.jar -> build/bdist.macosx-10.6-i386/egg
      creating build/bdist.macosx-10.6-i386/egg/simplejson
      copying build/lib/simplejson/_init_.py -> build/bdist.macosx-10.6-i386/egg/simplejson
      copying build/lib/simplejson/_speedups.c -> build/bdist.macosx-10.6-i386/egg/simplejson
      copying build/lib/simplejson/decoder.py -> build/bdist.macosx-10.6-i386/egg/simplejson
      copying build/lib/simplejson/encoder.py -> build/bdist.macosx-10.6-i386/egg/simplejson
      copying build/lib/simplejson/LICENSE.txt -> build/bdist.macosx-10.6-i386/egg/simplejson
      copying build/lib/simplejson/scanner.py -> build/bdist.macosx-10.6-i386/egg/simplejson
      copying build/lib/simplejson/tool.py -> build/bdist.macosx-10.6-i386/egg/simplejson
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/_init.py to __init_.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/datafile.py to datafile.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/io.py to io.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/ipc.py to ipc.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/protocol.py to protocol.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/schema.py to schema.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/tool.py to tool.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/avro/txipc.py to txipc.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/simplejson/_init.py to __init_.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/simplejson/decoder.py to decoder.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/simplejson/encoder.py to encoder.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/simplejson/scanner.py to scanner.pyc
      byte-compiling build/bdist.macosx-10.6-i386/egg/simplejson/tool.py to tool.pyc
      creating build/bdist.macosx-10.6-i386/egg/EGG-INFO
      installing scripts to build/bdist.macosx-10.6-i386/egg/EGG-INFO/scripts
      running install_scripts
      running build_scripts
      creating build/scripts-2.5
      copying and adjusting ./scripts/avro -> build/scripts-2.5
      changing mode of build/scripts-2.5/avro from 644 to 755
      creating build/bdist.macosx-10.6-i386/egg/EGG-INFO/scripts
      copying build/scripts-2.5/avro -> build/bdist.macosx-10.6-i386/egg/EGG-INFO/scripts
      changing mode of build/bdist.macosx-10.6-i386/egg/EGG-INFO/scripts/avro to 755
      copying avro.egg-info/PKG-INFO -> build/bdist.macosx-10.6-i386/egg/EGG-INFO
      copying avro.egg-info/SOURCES.txt -> build/bdist.macosx-10.6-i386/egg/EGG-INFO
      copying avro.egg-info/dependency_links.txt -> build/bdist.macosx-10.6-i386/egg/EGG-INFO
      copying avro.egg-info/requires.txt -> build/bdist.macosx-10.6-i386/egg/EGG-INFO
      copying avro.egg-info/top_level.txt -> build/bdist.macosx-10.6-i386/egg/EGG-INFO
      zip_safe flag not set; analyzing archive contents...
      creating 'dist/avro-AVRO_VERSION-py2.5.egg' and adding 'build/bdist.macosx-10.6-i386/egg' to it
      removing 'build/bdist.macosx-10.6-i386/egg' (and everything under it)
      Processing avro-AVRO_VERSION-py2.5.egg
      Copying avro-AVRO_VERSION-py2.5.egg to /Library/Python/2.5/site-packages
      Adding avro AVRO-VERSION to easy-install.pth file
      Installing avro script to /usr/local/bin

      Installed /Library/Python/2.5/site-packages/avro-AVRO_VERSION-py2.5.egg
      Processing dependencies for avro==AVRO-VERSION
      Searching for python-snappy
      Reading http://pypi.python.org/simple/python-snappy/
      Reading http://github.com/andrix/python-snappy
      Best match: python-snappy 0.3.2
      Downloading http://pypi.python.org/packages/source/p/python-snappy/python-snappy-0.3.2.tar.gz#md5=94ec3eb54a780fac3b15a6c141af973f
      Processing python-snappy-0.3.2.tar.gz
      Running python-snappy-0.3.2/setup.py q bdist_egg --dist-dir /tmp/easy_install-c1VYAq/python-snappy-0.3.2/egg-dist-tmp_CpoEp
      cc1plus: warning: command line option "-Wstrict-prototypes" is valid for C/ObjC but not for C++
      snappymodule.cc:31:22: error: snappy-c.h: No such file or directory
      snappymodule.cc: In function 'PyObject* snappy__compress(PyObject*, PyObject*)':
      snappymodule.cc:62: error: 'snappy_status' was not declared in this scope
      snappymodule.cc:62: error: expected `;' before 'status'
      snappymodule.cc:75: error: 'snappy_max_compressed_length' was not declared in this scope
      snappymodule.cc:79: error: 'status' was not declared in this scope
      snappymodule.cc:79: error: 'snappy_compress' was not declared in this scope
      snappymodule.cc:81: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc: In function 'PyObject* snappy__uncompress(PyObject*, PyObject*)':
      snappymodule.cc:107: error: 'snappy_status' was not declared in this scope
      snappymodule.cc:107: error: expected `;' before 'status'
      snappymodule.cc:120: error: 'status' was not declared in this scope
      snappymodule.cc:120: error: 'snappy_uncompressed_length' was not declared in this scope
      snappymodule.cc:121: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc:128: error: 'snappy_uncompress' was not declared in this scope
      snappymodule.cc:129: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc: In function 'PyObject* snappy__is_valid_compressed_buffer(PyObject*, PyObject*)':
      snappymodule.cc:151: error: 'snappy_status' was not declared in this scope
      snappymodule.cc:151: error: expected `;' before 'status'
      snappymodule.cc:156: error: 'status' was not declared in this scope
      snappymodule.cc:156: error: 'snappy_validate_compressed_buffer' was not declared in this scope
      snappymodule.cc:157: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc: At global scope:
      snappymodule.cc:41: warning: '_state' defined but not used
      cc1plus: warning: command line option "-Wstrict-prototypes" is valid for C/ObjC but not for C++
      snappymodule.cc:31:22: error: snappy-c.h: No such file or directory
      snappymodule.cc: In function 'PyObject* snappy__compress(PyObject*, PyObject*)':
      snappymodule.cc:62: error: 'snappy_status' was not declared in this scope
      snappymodule.cc:62: error: expected `;' before 'status'
      snappymodule.cc:75: error: 'snappy_max_compressed_length' was not declared in this scope
      snappymodule.cc:79: error: 'status' was not declared in this scope
      snappymodule.cc:79: error: 'snappy_compress' was not declared in this scope
      snappymodule.cc:81: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc: In function 'PyObject* snappy__uncompress(PyObject*, PyObject*)':
      snappymodule.cc:107: error: 'snappy_status' was not declared in this scope
      snappymodule.cc:107: error: expected `;' before 'status'
      snappymodule.cc:120: error: 'status' was not declared in this scope
      snappymodule.cc:120: error: 'snappy_uncompressed_length' was not declared in this scope
      snappymodule.cc:121: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc:128: error: 'snappy_uncompress' was not declared in this scope
      snappymodule.cc:129: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc: In function 'PyObject* snappy__is_valid_compressed_buffer(PyObject*, PyObject*)':
      snappymodule.cc:151: error: 'snappy_status' was not declared in this scope
      snappymodule.cc:151: error: expected `;' before 'status'
      snappymodule.cc:156: error: 'status' was not declared in this scope
      snappymodule.cc:156: error: 'snappy_validate_compressed_buffer' was not declared in this scope
      snappymodule.cc:157: error: 'SNAPPY_OK' was not declared in this scope
      snappymodule.cc: At global scope:
      snappymodule.cc:41: warning: '_state' defined but not used
      lipo: can't open input file: /var/tmp//ccbd0SLZ.out (No such file or directory)
      error: Setup script exited with error: command 'gcc-4.2' failed with exit status 1

      1. AVRO-981.patch
        0.4 kB
        Russell Jurney
      2. AVRO-981.patch
        0.7 kB
        Juan Manuel Caicedo
      3. AVRO-981.patch
        1 kB
        Miki Tebeka
      4. AVRO-981.patch
        1 kB
        Miki Tebeka

        Issue Links

          Activity

          Hide
          Miki Tebeka added a comment -

          Committed in revision 1395098.

          Show
          Miki Tebeka added a comment - Committed in revision 1395098.
          Hide
          Miki Tebeka added a comment -

          Catching ImportError

          Show
          Miki Tebeka added a comment - Catching ImportError
          Hide
          Miki Tebeka added a comment -

          Making snappy optional

          Show
          Miki Tebeka added a comment - Making snappy optional
          Hide
          Miki Tebeka added a comment -

          The patch is not good enough IMO, there are some code changes required as well. Attaching a patch, if it looks OK I'll commit.

          Show
          Miki Tebeka added a comment - The patch is not good enough IMO, there are some code changes required as well. Attaching a patch, if it looks OK I'll commit.
          Hide
          Doug Cutting added a comment -

          Thanks, Juan.

          This patch looks good to me. I'll commit it unless someone objects.

          Show
          Doug Cutting added a comment - Thanks, Juan. This patch looks good to me. I'll commit it unless someone objects.
          Hide
          Juan Manuel Caicedo added a comment -

          Hello,

          I'm attaching the patch the makes python-snappy a recommended dependency.

          Show
          Juan Manuel Caicedo added a comment - Hello, I'm attaching the patch the makes python-snappy a recommended dependency.
          Hide
          Doug Cutting added a comment -

          Juan> Please let me know if you think that this is a good solution. I'll be happy to send a patch file if you like.

          This seems to be the preferred solution. Can you please provide such a patch? Thanks!

          Show
          Doug Cutting added a comment - Juan> Please let me know if you think that this is a good solution. I'll be happy to send a patch file if you like. This seems to be the preferred solution. Can you please provide such a patch? Thanks!
          Hide
          Miki Tebeka added a comment -

          IMO the best solution is to have snappy as "recommended" and have the code conditionally import it.

          Show
          Miki Tebeka added a comment - IMO the best solution is to have snappy as "recommended" and have the code conditionally import it.
          Hide
          Juan Manuel Caicedo added a comment -

          Another option is to modify python-snappy so that it installs snappy as a static library. pyzmq does that, although the code seems to be a bit complicated and I thought that the 'recommended' approach was simple and good enough.

          Show
          Juan Manuel Caicedo added a comment - Another option is to modify python-snappy so that it installs snappy as a static library. pyzmq does that, although the code seems to be a bit complicated and I thought that the 'recommended' approach was simple and good enough.
          Hide
          Russell Jurney added a comment -

          This sounds reasonable to me - as long as we make a loud warning. Snappy compression seems very common from discussions on the lists, and this might lead to further confusion when python avro doesn't work reading a snappy avro file. Hopefully that error message is informative as well.

          Show
          Russell Jurney added a comment - This sounds reasonable to me - as long as we make a loud warning. Snappy compression seems very common from discussions on the lists, and this might lead to further confusion when python avro doesn't work reading a snappy avro file. Hopefully that error message is informative as well.
          Hide
          Juan Manuel Caicedo added a comment -

          Hello,

          I have an alternative solution for this problem: declare python-snappy as a 'recommended' dependency. By doing that it is no longer becomes necessary to have the snappy library installed, however, if someone states explicitly that wants to use that feature, the installer will require python-snappy.

          This is already supported by setuptools and the change in the code is trivial. You can see the changes in a sample GitHub project that I created:

          https://github.com/cavorite/python-avro/commit/ee338c88900fd73d88f08dfe4adbd4cf2416afbd

          Please let me know if you think that this is a good solution. I'll be happy to send a patch file if you like.

          Show
          Juan Manuel Caicedo added a comment - Hello, I have an alternative solution for this problem: declare python-snappy as a 'recommended' dependency. By doing that it is no longer becomes necessary to have the snappy library installed, however, if someone states explicitly that wants to use that feature, the installer will require python-snappy. This is already supported by setuptools and the change in the code is trivial. You can see the changes in a sample GitHub project that I created: https://github.com/cavorite/python-avro/commit/ee338c88900fd73d88f08dfe4adbd4cf2416afbd Please let me know if you think that this is a good solution. I'll be happy to send a patch file if you like.
          Hide
          Russell Jurney added a comment -

          I'm not too familiar with Python packaging, but ideally snappy and python-snappy would be auto-installed by python-avro. I don't want to have to install anything else - I want the install of python-avro to do it for me.

          Short of that, loudly stating that snappy and then python-snappy are required would help.

          The python-snappy guys look to be helping make the snappy dependence explicit, too.

          Thanks

          Show
          Russell Jurney added a comment - I'm not too familiar with Python packaging, but ideally snappy and python-snappy would be auto-installed by python-avro. I don't want to have to install anything else - I want the install of python-avro to do it for me. Short of that, loudly stating that snappy and then python-snappy are required would help. The python-snappy guys look to be helping make the snappy dependence explicit, too. Thanks
          Hide
          Doug Cutting added a comment -

          > snappy and python-snappy are dependencies that are not included in the project, and should be

          How do you mean they should they be included? Do you mean we should mention them in the top-level BUILD.txt or something else?

          Do you still feel we should remove them from setup.py?

          Thanks!

          Show
          Doug Cutting added a comment - > snappy and python-snappy are dependencies that are not included in the project, and should be How do you mean they should they be included? Do you mean we should mention them in the top-level BUILD.txt or something else? Do you still feel we should remove them from setup.py? Thanks!
          Hide
          Russell Jurney added a comment -

          Resolution of https://github.com/andrix/python-snappy/issues/7#issuecomment-3633856 calls this patch into question. Installing http://code.google.com/p/snappy/downloads/list makes python-snappy build, which should fix this issue.

          However, it seems that snappy, and python-snappy? are dependencies that are not included in the project, and should be.

          Show
          Russell Jurney added a comment - Resolution of https://github.com/andrix/python-snappy/issues/7#issuecomment-3633856 calls this patch into question. Installing http://code.google.com/p/snappy/downloads/list makes python-snappy build, which should fix this issue. However, it seems that snappy, and python-snappy? are dependencies that are not included in the project, and should be.
          Hide
          Russell Jurney added a comment -

          This patch removes snappy from setup.py, which allows avro to build, install and function on mac os x.

          Show
          Russell Jurney added a comment - This patch removes snappy from setup.py, which allows avro to build, install and function on mac os x.
          Hide
          Russell Jurney added a comment -

          This is a serious bug.

          Show
          Russell Jurney added a comment - This is a serious bug.
          Hide
          Russell Jurney added a comment -
          Show
          Russell Jurney added a comment - See related issue: https://github.com/andrix/python-snappy/issues/7
          Hide
          Russell Jurney added a comment -

          That would be great.

          Show
          Russell Jurney added a comment - That would be great.
          Hide
          Joe Crobak added a comment -

          We could file a fix upstream with python-snappy to compile the snappy library if it's not there.

          Secondly, looking through the code, it looks like snappy is meant to be an optional dependency, e.g.:

          lang/py/src/avro/datafile.py
          try:
            import snappy
          except:
            pass # fail later if snappy is used
          

          With a few minor changes, it seems like python-snappy could be removed as an explicit dependency in setup.py.

          Show
          Joe Crobak added a comment - We could file a fix upstream with python-snappy to compile the snappy library if it's not there. Secondly, looking through the code, it looks like snappy is meant to be an optional dependency, e.g.: lang/py/src/avro/datafile.py try : import snappy except: pass # fail later if snappy is used With a few minor changes, it seems like python-snappy could be removed as an explicit dependency in setup.py.
          Hide
          Ken Krugler added a comment -

          Has anyone found a workaround for this issue?

          I've got a client who's planning to consume our Avro-based results using Python, and asking him to install homebrew so he can install Snappy so he can install the Avro Python support (which doesn't yet work for me) so he can access the data isn't matching up well with my recommendation to use Avro as a good cross-language solution.

          Show
          Ken Krugler added a comment - Has anyone found a workaround for this issue? I've got a client who's planning to consume our Avro-based results using Python, and asking him to install homebrew so he can install Snappy so he can install the Avro Python support (which doesn't yet work for me) so he can access the data isn't matching up well with my recommendation to use Avro as a good cross-language solution.

            People

            • Assignee:
              Juan Manuel Caicedo
              Reporter:
              Russell Jurney
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development