[ARROW-5195] [Python] read_csv ignores null_values on string types

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.14.0
• Component/s: C++, Python
• Labels: pull-request-available
• Environment: Python 3.6, PyArrow 0.13.0, AWS linux, debian-slim in docker
• External issue URL: https://github.com/apache/arrow/issues/4184
• Assignee: Antoine Pitrou
• Reporter: Scott Burns
• Created: 22/Apr/19 20:59
• Updated: 11/Jan/23 07:38
• Resolved: 23/Apr/19 15:20

Description

Let's write a simple CSV with NULL values in a string column:

with open('foo.csv', 'w') as fobj:
    fobj.write('col1,col2\n1,value\n2,NULL')
table = csv.read_csv('foo.csv')
table.column('col2').null_count # => 0

 
table.column('col2').null_count will be 0, I think it should be 1. Passing in ConvertOptions(null_values=["NULL"]) doesn't help.

 

Note that pandas.read_csv parses these NULLs correctly so I have a workaround available.

But I'd prefer to natively read CSV from pyarrow if possible

Issue Links

• links to: GitHub Pull Request #4188