[ARROW-8802] [C++][Dataset] Schema metadata are lost when reading a subset of columns

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: C++
• Labels: dataset dataset-dask-integration pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24947
• Assignee: Francois Saint-Jacques
• Reporter: Joris Van den Bossche
• Created: 14/May/20 15:44
• Updated: 11/Jan/23 08:02
• Resolved: 18/Jun/20 15:32

Description

Python example:

import pandas as pd     
import pyarrow.dataset as ds                                                                                                                                                                              

df = pd.DataFrame({'a': [1, 2, 3]})  
df.to_parquet("test_metadata.parquet")  

dataset = ds.dataset("test_metadata.parquet")                                                                                                                                                             

gives:

>>> dataset.to_table().schema 
a: int64
  -- field metadata --
  PARQUET:field_id: '1'
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 397
ARROW:schema: '/////4ACAAAQAAAAAAAKAA4ABgAFAAgACgAAAAABAwAQAAAAAAAKAAwAAA' + 806

>>> dataset.to_table(columns=['a']).schema 
a: int64
  -- field metadata --
  PARQUET:field_id: '1'

So when specifying a subset of the columns, the additional metadata entries are lost (while those can still be informative, eg for conversion to pandas)

Issue Links

• links to: GitHub Pull Request #7474