[ARROW-8983] [Python] Downloading sources of pyarrow and its requirements from pypi takes several minutes starting from 0.16.0

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.16.0, 0.17.0, 0.17.1
• External issue URL: https://github.com/apache/arrow/issues/25105
• Assignee: Unassigned
• Reporter: Valentyn Tymofieiev
• Created: 29/May/20 17:56
• Updated: 11/Jan/23 08:03
• Resolved: 05/Jun/20 17:07

Description

It appears that

python -m pip download --dest /tmp pyarrow==0.17.1 --no-binary :all:

takes several minutes to execute.

There seems to be an increase in runtime starting from 0.16.0: on Python 2

python -m pip download --dest /tmp pyarrow==0.15.1 --no-binary :all:

appears to be somewhat faster, but the same command is still slow on Py3.

The command is stuck for a while with "Installing build dependencies ... ", and increased CPU usage.

The intent of this command is to download source tarball for a package and its dependencies.

Some investigation was started on the mailing list: https://lists.apache.org/thread.html/r9baa48a9d1517834c285f0f238f29fcf54405cb7cf1e681314239d7f%40%3Cdev.arrow.apache.org%3E