[ARROW-15312] [R][C++] filtering a Parquet dataset with is.na() misses some rows

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 6.0.1
• Fix Version/s: 8.0.0
• Component/s: R
• Labels: dataset pull-request-available
• Environment: R 4.1.2 on Windows ; arrow 6.0.1 ; dplyr 1.0.7
• External issue URL: https://github.com/apache/arrow/issues/20068

Language

C++ R

• Assignee: Jonathan Keane
• Reporter: Pierre Gramme
• Created: 12/Jan/22 16:21
• Updated: 11/Jan/23 11:36
• Resolved: 22/Apr/22 11:51

Description

Hi !

I just found an issue when querying an Arrow dataset with dplyr, filtering on is.na(...)

It seems linked to columns containing only one distinct value and some NA's.

Can you also reproduce the following?

 

  library(arrow)
  library(dplyr)
  
  ds_path = "test-arrow-na"
  df = tibble(x=1:3, y=c(0L, 0L, NA_integer_), z=c(0L, 1L, NA_integer_))
  
  df %>% arrow::write_dataset(ds_path)
  
  # OK: Collect then filter: returns row 3, as expected
  arrow::open_dataset(ds_path) %>% collect() %>% filter(is.na(y))

  # ERROR: Filter then collect (on y) returns a tibble with no row
  arrow::open_dataset(ds_path) %>% filter(is.na(y)) %>% collect()
  
  # OK: Filter then collect (on z) returns row 3, as expected
  arrow::open_dataset(ds_path) %>% filter(is.na(z)) %>% collect() 

 

Thanks

Pierre

Issue Links

• relates to: ARROW-14725 [C++][Compute] Extract Expression simplification passes to an extensible registry
• relates to: ARROW-12659 [C++][Compute] Support SimplifyWithGuarantee(is_null(foo), is_valid(foo))
• links to: apache/arrow PR#12891
• links to: GitHub Pull Request #12950