[HADOOP-17522] Looking to build Hadoop but start-build-env.sh errors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: build
• Assignee: Unassigned
• Reporter: Adam Roberts
• Created: 11/Feb/21 11:37
• Updated: 11/Feb/21 15:35
• Resolved: 11/Feb/21 15:35

Description

Hey everyone, we're trying to build Hadoop from source and in building.txt we see 

 

The easiest way to get an environment with all the appropriate tools is by means
of the provided Docker config.
This requires a recent version of docker (1.4.1 and higher are known to work).

On Linux / Mac:
Install Docker and run this command:

$ ./start-build-env.sh

 

However, and admittedly this is on 3.1.4, which I want to build - 

 

 

Step 26/36 : RUN pip2 install     configparser==4.0.2     pylint==1.9.2{{ --> Running in acf678bc8314}}Collecting configparser==4.0.2  Downloading https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whlCollecting pylint==1.9.2  Downloading https://files.pythonhosted.org/packages/f2/95/0ca03c818ba3cd14f2dd4e95df5b7fa232424b7fc6ea1748d27f293bc007/pylint-1.9.2-py2.py3-none-any.whl (690kB)Collecting singledispatch; python_version < "3.4" (from pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whlCollecting isort>=4.2.5 (from pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/a2/f7/f50fc9555dc0fe2dc1e7f69d93f71961d052857c296cad0fb6d275b20008/isort-5.7.0.tar.gz (169kB)Collecting astroid<2.0,>=1.6 (from pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/8b/29/0f7ec6fbf28a158886b7de49aee3a77a8a47a7e24c60e9fd6ec98ee2ec02/astroid-1.6.6-py2.py3-none-any.whl (305kB)Collecting backports.functools-lru-cache; python_version == "2.7" (from pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/da/d1/080d2bb13773803648281a49e3918f65b31b7beebf009887a529357fd44a/backports.functools_lru_cache-1.6.1-py2.py3-none-any.whlCollecting mccabe (from pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whlCollecting six (from pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whlCollecting enum34>=1.1.3; python_version < "3.4" (from astroid<2.0,>=1.6>pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/6f/2c/a9386903ece2ea85e9807e0e062174dc26fdce8b05f216d00491be29fad5/enum34-1.1.10-py2-none-any.whlCollecting wrapt (from astroid<2.0,>=1.6->pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/82/f7/e43cefbe88c5fd371f4cf0cf5eb3feccd07515af9fd6cf7dbf1d1793a797/wrapt-1.12.1.tar.gzCollecting lazy-object-proxy (from astroid<2.0,>=1.6->pylint==1.9.2)  Downloading https://files.pythonhosted.org/packages/69/fc/79080e582c17b7f45ac0bb4a13b0260992dcc2519b5a29ac6cde3e81b6fa/lazy_object_proxy-1.5.2-cp27-cp27mu-manylinux1_x86_64.whl (52kB)Building wheels for collected packages: isort, wrapt  Running setup.py bdist_wheel for isort: started  Running setup.py bdist_wheel for isort: finished with status 'error'  Complete output from command /usr/bin/python u -c "import setuptools, tokenize;_file='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), __file, 'exec'))" bdist_wheel -d /tmp/tmp60t8Japip-wheel --python-tag cp27:  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'    warnings.warn(msg)  running bdist_wheel  running build  running build_py  creating build  creating build/lib.linux-x86_64-2.7  creating build/lib.linux-x86_64-2.7/isort  copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort  copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort  copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort  copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort  copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort  copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort  copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort  copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort  copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort  copying isort/files.py -> build/lib.linux-x86_64-2.7/isort  copying isort/format.py -> build/lib.linux-x86_64-2.7/isort  copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort  copying isort/place.py -> build/lib.linux-x86_64-2.7/isort  copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort  copying isort/main.py -> build/lib.linux-x86_64-2.7/isort  copying isort/output.py -> build/lib.linux-x86_64-2.7/isort  copying isort/core.py -> build/lib.linux-x86_64-2.7/isort  copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort  copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort  copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort  copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort  copying isort/io.py -> build/lib.linux-x86_64-2.7/isort  copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort  copying isort/main.py -> build/lib.linux-x86_64-2.7/isort  copying isort/api.py -> build/lib.linux-x86_64-2.7/isort  copying isort/init.py -> build/lib.linux-x86_64-2.7/isort  copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort  creating build/lib.linux-x86_64-2.7/isort/_future  copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future  copying isort/_future/init.py -> build/lib.linux-x86_64-2.7/isort/_future  creating build/lib.linux-x86_64-2.7/isort/_vendored  creating build/lib.linux-x86_64-2.7/isort/_vendored/toml  copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml  copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml  copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml  copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml  copying isort/_vendored/toml/init.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml  creating build/lib.linux-x86_64-2.7/isort/deprecated  copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated  copying isort/deprecated/init.py -> build/lib.linux-x86_64-2.7/isort/deprecated  creating build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  copying isort/stdlibs/init.py -> build/lib.linux-x86_64-2.7/isort/stdlibs  error: can't copy 'isort/stdlibs': doesn't exist or not a regular file    ----------------------------------------  Failed building wheel for isort  Running setup.py clean for isort  Running setup.py bdist_wheel for wrapt: started  Running setup.py bdist_wheel for wrapt: finished with status 'done'  Stored in directory: /root/.cache/pip/wheels/b1/c2/ed/d62208260edbd3fa7156545c00ef966f45f2063d0a84f8208aSuccessfully built wraptFailed to build isortInstalling collected packages: configparser, six, singledispatch, isort, enum34, wrapt, backports.functools-lru-cache, lazy-object-proxy, astroid, mccabe, pylint  Running setup.py install for isort: started    Running setup.py install for isort: finished with status 'error'    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), __file, 'exec'))" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile:    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'      warnings.warn(msg)    running install    running build    running build_py    creating build    creating build/lib.linux-x86_64-2.7    creating build/lib.linux-x86_64-2.7/isort    copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort    copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort    copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort    copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort    copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort    copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort    copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort    copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort    copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort    copying isort/files.py -> build/lib.linux-x86_64-2.7/isort    copying isort/format.py -> build/lib.linux-x86_64-2.7/isort    copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort    copying isort/place.py -> build/lib.linux-x86_64-2.7/isort    copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort    copying isort/main.py -> build/lib.linux-x86_64-2.7/isort    copying isort/output.py -> build/lib.linux-x86_64-2.7/isort    copying isort/core.py -> build/lib.linux-x86_64-2.7/isort    copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort    copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort    copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort    copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort    copying isort/io.py -> build/lib.linux-x86_64-2.7/isort    copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort    copying isort/main.py -> build/lib.linux-x86_64-2.7/isort    copying isort/api.py -> build/lib.linux-x86_64-2.7/isort    copying isort/init.py -> build/lib.linux-x86_64-2.7/isort    copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort    creating build/lib.linux-x86_64-2.7/isort/_future    copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future    copying isort/_future/init.py -> build/lib.linux-x86_64-2.7/isort/_future    creating build/lib.linux-x86_64-2.7/isort/_vendored    creating build/lib.linux-x86_64-2.7/isort/_vendored/toml    copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml    copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml    copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml    copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml    copying isort/_vendored/toml/init.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml    creating build/lib.linux-x86_64-2.7/isort/deprecated    copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated    copying isort/deprecated/init.py -> build/lib.linux-x86_64-2.7/isort/deprecated    creating build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    copying isort/stdlibs/init.py -> build/lib.linux-x86_64-2.7/isort/stdlibs    error: can't copy 'isort/stdlibs': doesn't exist or not a regular file        ----------------------------------------Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), __file_, 'exec'))" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-u9WX19/isort/

You are using pip version 8.1.1, however version 21.0.1 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

 

If I add that into the Dockerfile, I get

}}{{ ---> Running in 5d4c46c61d29Traceback (most recent call last):  File "/usr/local/bin/pip2", line 7, in <module>    from pip._internal.cli.main import main  File "/usr/local/lib/python2.7/dist-packages/pip/_internal/cli/main.py", line 60    sys.stderr.write(f"ERROR: {exc}")                                   ^SyntaxError: invalid syntax

Issue Links

• duplicates: HADOOP-17120 Fix failure of docker image creation due to pip2 install error