[ARROW-17829] [Python] Avoid pandas groupby deprecation warning write_to_dataset

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 10.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/33050
• Assignee: Alenka Frim
• Reporter: Joris Van den Bossche
• Created: 23/Sep/22 14:06
• Updated: 11/Jan/23 11:55
• Resolved: 05/Oct/22 07:59

Description

I noticed the following warnings in our test builds:

opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_dataset.py::test_make_fragment
  /opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_dataset.py:197: FutureWarning: In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.
    for part, chunk in df_d.groupby(["color"]):

opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_dataset.py::test_legacy_write_to_dataset_drops_null
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/parquet/test_pandas.py::test_write_to_dataset_pandas_preserve_extensiondtypes[True]
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/parquet/test_pandas.py::test_write_to_dataset_pandas_preserve_index[True]
  /opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/parquet/core.py:3326: FutureWarning: In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.
    for keys, subgroup in data_df.groupby(partition_keys):

I suppose those are coming from pandas 1.5.0. We should investigate whether this is something to fix in our code (or just in the tests)