Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.12.0
-
$ python3 --version
Python 3.6.6
$ pip3 freeze | grep -Ei 'pyarrow|pandas'
pandas==0.24.1
pyarrow==0.12.0
Description
When writing or reading files to or from paths that have special characters in them, (e.g., "#"), pyarrow returns an error:
OSError: Passed non-file path...
This is a consequence of the following line:
https://github.com/apache/arrow/blob/master/python/pyarrow/filesystem.py#L416
File-paths will be parsed as URIs, which will give strange results for filepaths like: "bad # actor.parquet":
ParseResult(scheme='', netloc='', path='/tmp/bad ', params='', query='', fragment='actor.parquet')
This is trivial to reproduce with the following code which uses the `pd.to_parquet` and `pd.read_parquet` interfaces:
import pandas as pd x = pd.DataFrame({"a": [1,2,3]}) x.to_parquet("bad # actor.parquet") x.read_parquet("bad # actor.parquet")
Attachments
Issue Links
- is duplicated by
-
ARROW-4311 [Python] Regression on pq.ParquetWriter incorrectly handling source string
- Closed
- supercedes
-
ARROW-4480 [Python] Drive letter removed when writing parquet file
- Closed
- links to