[ARROW-11464] [Python] pyarrow.parquet.read_pandas doesn't conform to its docs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0
• Component/s: Python
• Labels: pull-request-available
• Environment: latest
• External issue URL: https://github.com/apache/arrow/issues/27348
• Assignee: Joris Van den Bossche
• Reporter: Pac A. He
• Created: 01/Feb/21 22:15
• Updated: 11/Jan/23 08:19
• Resolved: 07/Apr/21 08:55

Description

The pyarrow.parquet.read_pandas implementation doesn't conform to its docs in at least these ways:

1. The docs state that a filesystem option can be provided, as it should be. Without this option I cannot read from S3, etc. The implementation, however, doesn't have this option! As such I currently cannot use it to read from S3!
2. The docs (not in the type definition!) state that the default value for use_legacy_dataset is False, whereas the implementation has a value of True. Actually, however, if a value of True is used, then I can't read a partitioned dataset at all, so in this case maybe its the doc that needs to change to True.

It looks to have been implemented and reviewed pretty carelessly.

 

Issue Links

• links to: GitHub Pull Request #9910