Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-13618

[R] Use Arrow engine for summarize() by default

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Done
    • None
    • 6.0.0
    • R

    Description

      ARROW-13344 enabled the dplyr verb summarise() to use the Arrow engine but kept this off by default, controlled by the arrow.debug option.

      Before this can be turned on by default, we should ensure that the following are all implemented:

      • a sufficient set of hash aggregate kernels and R aggregate function mappings to them, covering the vast majority of all aggregate functions that dplyr users call in summarise() (add any additional required ones to ARROW-13339)
      • support for a sufficient set of data types in aggregates
      • support for a sufficient set of data types in grouping columns
      • handling of NA and NaN values in aggregates and the na.rm option consistent with base R and dplyr (ARROW-13497 and possibly other issues)
      • handling of NA and NaN values in grouping columns consistent with dplyr
      • handling empty or bad input to summarise() (ARROW-13543)
      • many new tests to confirm equivalent results from a variety of group_by() %>% summarise() queries on data frames and on Arrow data
      • resolution of various related bugs

      Attachments

        Issue Links

          Activity

            People

              icook Ian Cook
              icook Ian Cook
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: