[ARROW-17672] [C++] Unrecognized filesystem type in URI: abfss://

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++, Parquet, Python
• External issue URL: https://github.com/apache/arrow/issues/32912
• Assignee: Unassigned
• Reporter: Prakhar Sandhu
• Created: 10/Sep/22 08:02
• Updated: 11/Jan/23 11:52

Description

I am running the below commands in databricks.

When I am trying to read a file which is stored in adls using pandas:

pip install adlfs 
import pandas as pd
data = pd.read_parquet("abfss://data.parquet", storage_options= {})

Then I got the below error: 

File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py", line 310, in read_parquet
return impl.read(path, columns=columns, **kwargs)
File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py", line 125, in read
path, columns=columns, **kwargs
File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line 1573, in read_table
ignore_prefixes=ignore_prefixes,
File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line 1434, in __init__
ignore_prefixes=ignore_prefixes)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line 667, in dataset
return _filesystem_dataset(source, **kwargs)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line 424, in _filesystem_dataset
fs, paths_or_selector = _ensure_single_source(source, filesystem)
File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line 371, in _ensure_single_source
filesystem, path = FileSystem.from_uri(path)
File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI: abfss://data.parquet