[IMPALA-11846] impala-shell packaging is broken with python 3.8 and setuptools 66.0.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Epic Color: ghx-label-1
• Assignee: Gergely Fürnstáhl
• Reporter: Gergely Fürnstáhl
• Created: 17/Jan/23 10:07
• Updated: 29/Jan/23 07:34
• Resolved: 25/Jan/23 00:22

Description

setuptools introduced a breaking change with 66.0.0 release affecting impala build:
https://setuptools.pypa.io/en/stable/history.html#breaking-changes

It affects building with python 3.8. setuptools package is part of the build system and installed with python and pip during an earlier step, the version number cannot be specified in the setup.py file.

Failed to build impala-shell
Installing collected packages: bitarray, configparser, kerberos, prettytable, six, sasl, sqlparse, thrift, pure-sasl, thrift-sasl, impala-shell
  Attempting uninstall: thrift
    Found existing installation: thrift 0.11.0
    Not uninstalling thrift at /home/gfurnstahl/Impala/toolchain/toolchain-packages-gcc10.4.0/thrift-0.11.0-p5/python/lib/python2.7/site-packages, outside environment /home/gfurnstahl/Impala/shell/build/py2_venv
    Can't uninstall 'thrift'. No files were found to uninstall.
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      running egg_info
      /home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/dist.py:543: UserWarning: The version specified ('install-test') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
        warnings.warn(
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-qi4d4ya9/setup.py", line 136, in <module>
          setup(
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.ensure_finalized()
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
          self.finalize_options()
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 219, in finalize_options
          parsed_version = parse_version(self.egg_version)
        File "/home/gfurnstahl/Impala/shell/build/py3_venv/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
          raise InvalidVersion(f"Invalid version: '{version}'")
      pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: 'install-test'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output. 

Jenkins jobs does not seem to be affected due to using python 3.5 and installing older version of setuptools.

19:13:52 DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
19:13:52 Requirement already satisfied: setuptools>=36.8.0 in ./build/py3_venv/lib/python3.5/site-packages (from impala-shell===install-test) (50.3.2) 

A solution could be to use PEP440 compilant name for the test package (e.g. replace install-test with 1.0.0.dev1), or extend the packaging using PEP518 and pyproject.toml to specify setuptools version:

https://peps.python.org/pep-0518/

Followup Jira to track improvements in the stability of python env:

https://issues.apache.org/jira/browse/IMPALA-11849