Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-13627

[C++] ScalarAggregateOptions don't make sense (in hash aggregation)

    XMLWordPrintableJSON

Details

    Description

      R's aggregation functions have a na.rm argument that governs how missing data is handled. Assume x <- c(1, 2, NA, 3). sum(x, na.rm = TRUE) == 6 and sum(x, na.rm = FALSE) is NA because there is at least one missing value.

      The ScalarAggregateOptions have two options: skip_nulls and min_count. From what I can tell reading the source, in the context of sum(), skip_nulls affects whether each element of the Array is added to "count", and if count < min_count, you get a null value returned. So to get the expected behavior when calling "sum" on an Array, when na.rm = TRUE, we pass skip_nulls = false, min_count = 0. When na.rm = FALSE, we pass skip_nulls = true, min_count = length, the reasoning being that you return a null value unless all values are non-null (and count == length). See https://github.com/apache/arrow/blob/master/r/R/compute.R#L125-L130

      This doesn't really work in the query engine, though. We don't know how many rows are in the data to set an appropriate min_count to get the expected behavior--the dataset being queried may have filtering. And when doing hash aggregation, each group may have a different number of rows.

      Attachments

        Issue Links

          Activity

            People

              lidavidm David Li
              npr Neal Richardson
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h