Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-11925

[R] Add `between` method for arrow_dplyr_query

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 4.0.0
    • R

    Description

      Would you consider a PR to add a between method for `arrow_dplyr_query` objects? Even something implemented directly in R harnesses the arrow speed. Here is what I am thinking:

      Typical usage of `between`:

       

      library(dplyr)
      library(arrow)
      iris %>% filter(between(Petal.Length, 1, 1.1))

       

       Here is a mocked up version of the method:

       

      between_mock <- function(x, left, right) {
      if (length(left) != 1) {
       rlang::abort("`left` must be length 1")
       }
       if (length(right) != 1) {
       rlang::abort("`right` must be length 1")
       }x >= left & x <= right
      }

      I think because `dplyr` uses C++ to efficiently do this, `between` doesn't work out of the box:

      open_dataset("nyc-taxi", partitioning = "year") %>% 
       filter(year == 2014) %>% 
       select(year, fare_amount) %>% 
       filter(between(fare_amount, 10, 11)) %>% 
       collect() 
      
      Error: Filter expression not supported for Arrow Datasets: between(fare_amount, 10, 11)
      Call collect() first to pull data into R.
      In addition: Warning message:
       between() called on numeric vector with S3 class 
      Backtrace:
       x
      1. +-[ `%>%`(...) ]
      2. +-[ dplyr::collect(...) ]
      3. +-[ dplyr::filter(...) ]
      4. \-arrow:::filter.arrow_dplyr_query(...)

      But even my simple implementation works fine:

      open_dataset("nyc-taxi", partitioning = "year") %>% 
       filter(year == 2014) %>% 
       select(year, fare_amount) %>% 
       filter(between_mock(fare_amount, 10, 11)) %>% 
       collect() 

       

       

       

      Attachments

        Issue Links

          Activity

            People

              boshek Sam Albers
              boshek Sam Albers
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 50m
                  1h 50m