[ARROW-9814] [Python] Crash in test_parquet.py::test_read_partitioned_directory_s3fs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: C++, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25856
• Assignee: Antoine Pitrou
• Reporter: Antoine Pitrou
• Created: 20/Aug/20 16:07
• Updated: 11/Jan/23 08:09
• Resolved: 09/Sep/20 07:48

Description

This seems to happen with some Minio versions, but is definitely a problem in Arrow.
The crash message says:

pyarrow/tests/test_parquet.py::test_read_partitioned_directory_s3fs[False] ../src/arrow/dataset/discovery.cc:188:  Check failed: relative.has_value() GetFileInfo() yielded path outside selector.base_dir

The underlying problem is that we pass a full URI for the selector base_dir (such as "s3://bucket/path.") and the S3 filesystem implementation then returns regular paths (such as "bucket/path/foo/bar").

I think we should do two things:
1) error out rather than crash (and include the path strings in the error message), which would be more user-friendly
2) fix the issue that full URIs are passed in base_dir

Issue Links

• links to: GitHub Pull Request #8139