[ARROW-13618] [R] Use Arrow engine for summarize() by default

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: R
• Labels: query-engine
• External issue URL: https://github.com/apache/arrow/issues/29259
• Assignee: Ian Cook
• Reporter: Ian Cook
• Created: 12/Aug/21 17:19
• Updated: 11/Jan/23 08:34
• Resolved: 29/Aug/22 14:14

Description

ARROW-13344 enabled the dplyr verb summarise() to use the Arrow engine but kept this off by default, controlled by the arrow.debug option.

Before this can be turned on by default, we should ensure that the following are all implemented:

• a sufficient set of hash aggregate kernels and R aggregate function mappings to them, covering the vast majority of all aggregate functions that dplyr users call in summarise() (add any additional required ones to ARROW-13339)
• support for a sufficient set of data types in aggregates
• support for a sufficient set of data types in grouping columns
• handling of NA and NaN values in aggregates and the na.rm option consistent with base R and dplyr (ARROW-13497 and possibly other issues)
• handling of NA and NaN values in grouping columns consistent with dplyr
• handling empty or bad input to summarise() (ARROW-13543)
• many new tests to confirm equivalent results from a variety of group_by() %>% summarise() queries on data frames and on Arrow data
• resolution of various related bugs

Issue Links

• is blocked by: ARROW-13497 [C++][R] FunctionOptions not used by aggregation nodes
• is blocked by: ARROW-13499 [R] Aggregation on expression doesn't NSE correctly
• is blocked by: ARROW-13543 [R] Handle summarize() with 0 arguments or no aggregate functions
• is blocked by: ARROW-13502 [R] Bindings for min/max aggregation
• is blocked by: ARROW-13550 [R] Support .groups argument to dplyr::summarize()
• is blocked by: ARROW-13501 [R] Bindings for count aggregation
• is blocked by: ARROW-13528 [R] Bindings for mean, var, sd aggregation
• is blocked by: ARROW-13772 [R] Binding for median() and quantile() aggregation functions
• is blocked by: ARROW-13777 [R] mutate after group_by should be ok as long as there are only scalar functions
• is blocked by: ARROW-13778 [R] Handle complex summarize expressions
• is blocked by: ARROW-13691 [C++] Add option to handle NAs to VarianceOptions
• is blocked by: ARROW-13740 [R] summarize() should not eagerly evaluate
• is blocked by: ARROW-13764 [C++] Implement ScalarAggregateOptions for count_distinct (grouped)
• relates to: ARROW-13344 [R] Initial bindings for ExecPlan/ExecNode