[ARROW-6327] [Python] Conversion of pandas.SparseArray columns in pandas.DataFrames to pyarrow.Table and back

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Won't Do
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/22705
• Assignee: Unassigned
• Reporter: Rok Mihevc
• Created: 22/Aug/19 20:08
• Updated: 11/Jan/23 07:46
• Resolved: 05/Aug/21 15:09

Description

We would like to convert sparse columns from Pandas to Arrow:

import numpy as np
import pandas
import pyarrow

arr = pandas.Series([1, 2, 3])
sparr = pandas.SparseArray(np.array([1, 0, 0], dtype='int64'))
df = pandas.DataFrame({'sparr': sparr, 'arr': arr})

table = pyarrow.table(df)
df == table.to_pandas()

I assume `pandas.SparseArray` is a 1D sparse COO Tensor that would map to `pyarrow.SparseTensorCOO`.