[ARROW-17641] [python] Deserializing ParseOptions does not set up invalid row handler correctly

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 9.0.0
• Fix Version/s: 10.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32883
• Assignee: Kai Fricke
• Reporter: Kai Fricke
• Created: 07/Sep/22 10:06
• Updated: 11/Jan/23 11:52
• Resolved: 08/Sep/22 07:14

Description

Serializing and deserializing a csv.ParseOptions object with an invalid_row_handler will render the handler unusable. This is likely because the setter is not called correctly in the _setstate_ method.

Reproduction script:

 

import cloudpickle
from pyarrow import csv


invalid_csv = """f1,f2
3,4
5,6
\x00\x00
7,8"""

source = "test.csv"
with open(source, "w") as f:
    f.write(invalid_csv)


def read_file(path, parse_options):
    # Uncomment this for a fix!
    # parse_options.invalid_row_handler = parse_options.invalid_row_handler

    with open(path, "rb") as f:
        return csv.read_csv(f, parse_options=parse_options)


parse_options = csv.ParseOptions(delimiter=",", invalid_row_handler=lambda i: "skip")

# Will succeed
print(read_file(source, parse_options=parse_options))

parse_options = cloudpickle.loads(cloudpickle.dumps(parse_options))

# Will fail
print(read_file(source, parse_options=parse_options))

 

Issue Links

• links to: GitHub Pull Request #14061