[ARROW-12762] [Python] ListType doesn't preserve field name after pickle and unpickle

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 5.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28504
• Assignee: Alessandro Molina
• Reporter: Juan Galvez
• Created: 12/May/21 20:03
• Updated: 11/Jan/23 08:28
• Resolved: 13/Jul/21 12:40

Description

Here is a small reproducer:

import pandas as pd
from pyspark.sql import SparkSession
import pyarrow.parquet as pq
import pickle

df = pd.DataFrame(
    {
        "A": [
            ["aa", "bb "],
            ["c"],
            ["d", "ee", "", "f"],
            ["ggg", "H"],
            [""],
        ]
    }
)

spark = SparkSession.builder.appName("GenSparkData").getOrCreate()
spark_df = spark.createDataFrame(df)
spark_df.write.parquet("list_str.pq", "overwrite")

ds = pq.ParquetDataset("list_str.pq")
assert pickle.loads(pickle.dumps(ds.schema)) == ds.schema # PASSES
assert pickle.loads(pickle.dumps(ds.schema.to_arrow_schema())) == ds.schema.to_arrow_schema() # FAILS

Issue Links

• links to: GitHub Pull Request #10704