[ARROW-17308] ValueError: Keyword 'validate_schema' is not yet supported with the new Dataset API

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Python
• Labels: dataset-parquet-legacy dataset-parquet-read
• External issue URL: https://github.com/apache/arrow/issues/32585
• Assignee: Unassigned
• Reporter: Abderrahmane Jaidi
• Created: 04/Aug/22 16:11
• Updated: 11/Jan/23 11:49
• Resolved: 20/Oct/22 12:31

Description

Documentation for PyArrow 6.x and 7.x both indicate that the `validate_schema` argument is supported in the `ParquetDataset` class. Yet passing that argument to an instance results in:

ValueError: Keyword 'validate_schema' is not yet supported with the new Dataset API

Code:

parquet_dataset = pyarrow.parquet.ParquetDataset(
    path_or_paths=paths,
    validate_schema=validate_schema,
    filesystem=filesystem,
    partitioning=partitioning,
    use_legacy_dataset=False,
)

Docs link:

https://arrow.apache.org/docs/6.0/python/generated/pyarrow.parquet.ParquetDataset.html

https://arrow.apache.org/docs/7.0/python/generated/pyarrow.parquet.ParquetDataset.html 

Issue Links

• duplicates: ARROW-8221 [Python][Dataset] Expose schema inference / validation options in the factory