[ARROW-8644] [Python] Dask integration tests failing due to change in not including partition columns

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24805
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 30/Apr/20 08:57
• Updated: 11/Jan/23 08:01
• Resolved: 05/May/20 21:43

Description

In ARROW-3861 (https://github.com/apache/arrow/pull/7050), I "fixed" a bug that the partition columns are always included even when the user did a manual column selection.

But apparently, this behaviour was being relied upon by dask. See the failing nightly integration tests: https://circleci.com/gh/ursa-labs/crossbow/11854?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

So the best option might be to just keep the "old" behaviour for the legacy ParquetDataset, when using the new datasets API (use_legacy_datasets=False), you get the new / correct behaviour.

Issue Links

• links to: GitHub Pull Request #7096