[ARROW-4452] [Python] Serializing sparse torch tensors

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21010
• Assignee: Philipp Moritz
• Reporter: Philipp Moritz
• Created: 01/Feb/19 20:15
• Updated: 11/Jan/23 07:34
• Resolved: 07/Jun/19 15:26

Description

Using the pytorch serialization handler on sparse Tensors:

import torch
i = torch.LongTensor([[0, 2], [1, 0], [1, 2]])
v = torch.FloatTensor([3,      4,      5    ])
tensor = torch.sparse.FloatTensor(i.t(), v, torch.Size([2,3]))

pyarrow.serialization.register_torch_serialization_handlers(pyarrow.serialization._default_serialization_context)

s = pyarrow.serialize(tensor, context=pyarrow.serialization._default_serialization_context) 

Produces this result:

TypeError: can't convert sparse tensor to numpy. Use Tensor.to_dense() to convert to a dense tensor first.

We should provide a way to serialize sparse torch tensors, especially now that we are getting support for sparse Tensors.

Issue Links

• links to: GitHub Pull Request #3542