[ARROW-16495] [Python] Scanner.count_rows() doesn't properly handle null expressions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.0.0
• Fix Version/s: 8.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/31861
• Assignee: Unassigned
• Reporter: Nick Riasanovsky
• Created: 06/May/22 17:21
• Updated: 11/Jan/23 11:44
• Resolved: 10/May/22 18:12

Description

Passing an expression filter with `is_null()` doesn't properly remove null values, when computing row counts. I have reproduced this with both strings and integer. Here is a reproducer.

 

 

 

df = pd.DataFrame({"C": pd.array([None, None, 1], dtype=pd.Int64Dtype())})
print(df)
df.to_parquet("test.pq")
 
# Create a dataset
dataset = ds.dataset("test.pq")
fragments = [f for f in dataset.get_fragments()]
#There should just be 1 fragment.
fragment = fragments[0]
# Get the null row count
expr = ds.field("C").is_null()
scanner = fragment.scanner(filter=expr)
print(scanner.count_rows())

 

 

I expect this print 2 as there are 2 NULL values.