[ARROW-9261] [Python][Packaging] S3FileSystem curl errors in manylinux wheels

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Packaging, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25356
• Assignee: Antoine Pitrou
• Reporter: Roee Shlomo
• Created: 28/Jun/20 18:57
• Updated: 11/Jan/23 08:05
• Resolved: 30/Jun/20 10:06

Description

https://issues.apache.org/jira/browse/ARROW-9109 introduced S3 support in manylinux wheels. However, when trying to use S3FileSystem it fails with

 

Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "pyarrow/_fs.pyx", line 597, in pyarrow._fs.FileSystem.open_input_stream
 File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
 File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: When reading information for key 'xxxxx' in bucket 'xxxxx': AWS Error [code 99]: curlCode: 77, Problem with the SSL CA cert (path? access rights?) with address

It seems like it can't find the SSL CA cert directory that is installed in the runtime machine (tested on Ubuntu 16.04 and Ubuntu 18.04). It always searches in /etc/pki/tls/certs/ca-bundle.crt probably because the wheels are built on centos, whereas in Ubuntu the path is /etc/ssl/certs/ca-certificates.crt and is different on other distributions.

Reproduce with:

virtualenv -p python3.8 arrowenv
source arrowenv/bin/activate
pip install --extra-index-url https://repo.fury.io/arrow-nightlies/ --pre pyarrow
python -c "from pyarrow.fs import S3FileSystem; fs = S3FileSystem(); fs.open_input_stream('mybucket/myfile')"

 

 

Issue Links

• links to: GitHub Pull Request #7580