[IMPALA-12117] Intermittent pip error when building shell's venv

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.3.0
• Fix Version/s: Impala 4.3.0
• Component/s: Infrastructure
• Epic Color: ghx-label-12
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 05/May/23 03:33
• Updated: 08/May/23 17:33
• Resolved: 08/May/23 17:31

Description

The all-build-options-ub2004 job intermittently hits this pip error when building the shell virtualenvs:

 

WARNING: Skipping page https://pypi.org/simple/thrift/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
ERROR: Could not find a version that satisfies the requirement thrift==0.16.0 (from impala-shell) (from versions: none)
ERROR: No matching distribution found for thrift==0.16.0

This error doesn't always apply to thrift. It can be any of the shell dependencies. It doesn't seem to impact the impala-python or impala-python3 virtualenv (which don't use prebuilt binaries).

Searching around, a theory on the forums is that this can happen when there are multiple pip installs running simultaneously writing to the same pip cache. Impala builds a python 2 virtualenv and a python 3 virtualenv simultaneously, so that seems like a plausible theory.

We should try using separate pip caches for the python 2 vs python 3 shell virtualenvs.