[ARROW-16904] [C++] min/max not deterministic if Parquet files have multiple row groups

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 8.0.0
• Fix Version/s: 9.0.0
• Component/s: R
• Labels: pull-request-available
• Environment: $ lsb_release -a ; No LSB modules are available. ; Distributor ID: Ubuntu ; Description: Ubuntu 20.04.4 LTS ; Release: 20.04 ; Codename: focal
• Flags: Important
• External issue URL: https://github.com/apache/arrow/issues/20300
• Assignee: Aldrin Montana
• Reporter: Robert On
• Created: 25/Jun/22 17:36
• Updated: 11/Jan/23 11:47
• Resolved: 11/Jul/22 13:15

Description

The following code produces non-deterministic result for getting the minimum value of a sequence of 1e5 and 1e6 integers.

sapply(1:100, function(x) {
  # create parquet file with a val column with numbers 1 to 100,000
  arrow::write_parquet(
    data.frame(val = 1:1e5), "test.parquet")
  # find minimum value
  arrow::open_dataset("test.parquet") %>%
    dplyr::summarise(min_val = min(val)) %>%
    dplyr::collect() %>% dplyr::pull(min_val)
}) %>% table()

sapply(1:100, function(x) {
  # create parquet file with a val column with numbers 1 to 1,000,000
  arrow::write_parquet(
    data.frame(val = 1:1e6), "test.parquet")
  # find minimum value
  arrow::open_dataset("test.parquet") %>%
    dplyr::summarise(min_val = min(val)) %>%
    dplyr::collect() %>% dplyr::pull(min_val)
}) %>% table()

The first 100 simulations using numbers 1 to 1e5 is able to find the minimum number (1) all 100 times.

The second 100 simulations using numbers 1 to 1e6 only finds the minimum number (1) 65 out of 100 times. It finds near multiples of 131073, 25, 8, and 2 times respectively.

. 1
100 
. 1 131073 262145 393217 
 65     25      8      2 

 

Issue Links

• relates to: ARROW-16807 [C++] count_distinct aggregates incorrectly across row groups
• links to: GitHub Pull Request #13509