[ARROW-5310] [Python] better error message on creating ParquetDataset from empty directory

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Python
• Labels: dataset dataset-parquet-read parquet
• External issue URL: https://github.com/apache/arrow/issues/16728
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 13/May/19 14:24
• Updated: 11/Jan/23 07:39
• Resolved: 04/May/20 23:43

Description

Currently, you get when path is an existing but empty directory:

>>> dataset = pq.ParquetDataset(path)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-16-346f72ae525e> in <module>
----> 1 dataset = pq.ParquetDataset(path)

~/scipy/repos/arrow/python/pyarrow/parquet.py in __init__(self, path_or_paths, filesystem, schema, metadata, split_row_groups, validate_schema, filters, metadata_nthreads, memory_map)
    989 
    990         if validate_schema:
--> 991             self.validate_schemas()
    992 
    993         if filters is not None:

~/scipy/repos/arrow/python/pyarrow/parquet.py in validate_schemas(self)
   1025                 self.schema = self.common_metadata.schema
   1026             else:
-> 1027                 self.schema = self.pieces[0].get_metadata().schema
   1028         elif self.schema is None:
   1029             self.schema = self.metadata.schema

IndexError: list index out of range

That could be a nicer error message.

Unless we actually want to allow this? (although I am not sure there are good use cases of empty directories to support this, because from an empty directory we cannot get any schema or metadata information?)
It is only failing when validating the schemas, so with validate_schema=False it actually returns a ParquetDataset object, just with an empty list for pieces and no schema. So it would be easy to not error when validating the schemas as well for this empty-directory case.

Issue Links

• depends upon: ARROW-8039 [Python][Dataset] Support using dataset API in pyarrow.parquet with a minimal ParquetDataset shim