[ARROW-15910] [Python] pyarrow.parquet.read_table either returns FileNotFound or ArrowInvalid

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 6.0.1, 7.0.0
• Component/s: Parquet, Python
• Environment: GCP JupyterLab notebooks
• External issue URL: https://github.com/apache/arrow/issues/31339
• Assignee: Unassigned
• Reporter: Callista Rogers
• Created: 10/Mar/22 17:14
• Updated: 11/Jan/23 11:40

Description

running below results in "GetFileIno() yielded path 'myBucket/features/MyParquet.parquet/year=2022/part-0019.snappy.parquet' which is outside base dir 'gs://myBucket/features/MyParquet.parquet/' "

import pyarrow.parquet as pq
import gcsfs
file_path="gs://myBucket/features/MyParquet.parquet/"
fs=gcsfs.GCSFileSystem()
table=pq.read_table(file_path,filesystem=fs) 

Removing the gs:// from file_path results in a FileNotFoundError. Any variation of / or // at the beginning of the path gives me the 'outside base dir' error.

I also ran the below and got valid results using both file_path patterns, so I know it finds the path just fine.

from pyarrow.fs import FileSelector, PyFileSystem, FSSpecHandler
filesys = PyFileSystem(FSSpecHandler(fs))
selector = FileSelector(file_path, recursive=True)
filesys.get_file_info(selector)