Details
-
Improvement
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Recent addition of the fastavro dependency is breaking python linter in macos. At least for some cases, because it requires a compiler.
Could we optionally depend on fastavro, and fallback to regular avro package?
Log:
> Task :beam-sdks-python:lintPy27
ERROR: invocation failed (exit code 1), logfile: /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
ERROR: actionid: py27-lint
msg: installpkg
cmdargs: ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python', '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip', 'install', '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from apache-beam==2.6.0.dev0) (1.11.0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
Requirement already satisfied: future<1.0.0,>=0.16.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from apache-beam==2.6.0.dev0) (0.16.0)
Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
Requirement already satisfied: setuptools>=18.0 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
Requirement already satisfied: enum34>=1.0.4 in ./target/.tox/py27-lint/lib/python2.7/site-packages (from grpcio<2,>=1.8->apache-beam==2.6.0.dev0) (1.1.6)
Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
Collecting requests>=2.7.0 (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
Collecting pyasn1>=0.1.7 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
Collecting pyasn1-modules>=0.0.5 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
Collecting idna<2.8,>=2.5 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
Collecting urllib3<1.24,>=1.21.1 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
Collecting certifi>=2017.4.17 (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
Building wheels for collected packages: apache-beam, fastavro, pyyaml
Running setup.py bdist_wheel for apache-beam: started
Running setup.py bdist_wheel for apache-beam: finished with status 'done'
Stored in directory: /Users/relax/Library/Caches/pip/wheels/a3/18/08/60ce57e27e8b04460419c9463b9b6da19a8aab9fb64afc612f
Running setup.py bdist_wheel for fastavro: started
Running setup.py bdist_wheel for fastavro: finished with status 'error'
Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;_file='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file_, 'exec'))" bdist_wheel -d /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-wheel-p6UGao --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.4-x86_64-2.7
creating build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_init_.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_main_.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
running build_ext
building 'fastavro._read' extension
creating build/temp.macosx-10.4-x86_64-2.7
creating build/temp.macosx-10.4-x86_64-2.7/fastavro
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for fastavro
Running setup.py clean for fastavro
Running setup.py bdist_wheel for pyyaml: started
Running setup.py bdist_wheel for pyyaml: finished with status 'done'
Stored in directory: /Users/relax/Library/Caches/pip/wheels/03/05/65/bdc14f2c6e09e82ae3e0f13d021e1b6b2481437ea2f207df3f
Successfully built apache-beam pyyaml
Failed to build fastavro
Installing collected packages: avro, crcmod, dill, fastavro, futures, grpcio, docopt, chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr, funcsigs, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pytz, pyyaml, pyvcf, typing, nose, pyhamcrest, apache-beam
Running setup.py install for fastavro: started
Running setup.py install for fastavro: finished with status 'error'
Complete output from command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;_file='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file_, 'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt --single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.4-x86_64-2.7
creating build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_schema_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_init_.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_read_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_validate_common.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_validation_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/_main_.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
running build_ext
building 'fastavro._read' extension
creating build/temp.macosx-10.4-x86_64-2.7
creating build/temp.macosx-10.4-x86_64-2.7/fastavro
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7 -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u -c "import setuptools, tokenize;_file='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file_, 'exec'))" install --record /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt --single-version-externally-managed --compile --install-headers /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro" failed with error code 1 in /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/
py27-lint installed: astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,configparser==3.5.0,crcmod==1.7,dill==0.2.6,enum34==1.1.6,flake8==3.5.0,future==0.16.0,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,pycodestyle==2.3.1,pyflakes==1.6.0,pylint==1.7.2,singledispatch==3.4.0.3,six==1.11.0,wrapt==1.10.11
___________________________________ summary ____________________________________
ERROR: py27-lint: InvocationError for command /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip install /Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test] (see /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log) (exited with code 1)