Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7772

[R][C++][Dataset] Unable to filter on date32 object with date64 scalar

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.16.0
    • 0.17.0
    • C++, R
    • version R version 3.6.2 (2019-12-12)
       os macOS Mojave 10.14.6
       system x86_64, darwin15.6.0

    Description

      I am trying to filter on a date column using `open_dataset()` and `dplyr::filter()`:

      library(arrow)
      library(dplyr)
      
      tmp <- tempfile()
      dir.create(tmp)
      df <- data.frame(date = Sys.Date())
      write_parquet(df, file.path(tmp, "file.parquet"))
      
      ds <- open_dataset(tmp)
      
      
      ds %>%
       filter(date > as.Date("2020-02-02")) %>%
       collect()
      

       

      This code crashes R with this error message:

      /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/hbtmp/apache-arrow-20200203-29929-1uoyri7/cpp/src/arrow/result.cc:28: ValueOrDie called on an error: NotImplemented: casting scalarsof type date64[ms] to type date32[day]
      0 arrow.so 0x0000000104461f1d _ZN5arrow4util7CerrLogD2Ev + 209
      1 arrow.so 0x0000000104461e3e _ZN5arrow4util7CerrLogD0Ev + 14
      2 arrow.so 0x0000000104461de6 _ZN5arrow4util8ArrowLogD1Ev + 34
      3 arrow.so 0x000000010436c57f ZN5arrow8internal14DieWithMessageERKNSt3_112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 63
      4 arrow.so 0x000000010436d384 ZNR5arrow6ResultINSt3_110shared_ptrINS_6ScalarEEEE10ValueOrDieEv + 192
      5 arrow.so 0x000000010426ee1a ZN5arrow7dataset23InsertImplicitCastsImpl4CastENSt3_110shared_ptrINS_8DataTypeEEEPNS3_INS0_10ExpressionEEE + 186
      6 arrow.so 0x000000010426e033 _ZN5arrow7dataset23InsertImplicitCastsImplclERKNS0_20ComparisonExpressionE + 757
      7 arrow.so 0x0000000104269de7 ZN5arrow7dataset15VisitExpressionINS0_23InsertImplicitCastsImplEEEDTclfp0_fp_EERKNS0_10ExpressionEOT + 317
      8 arrow.so 0x0000000104269c76 _ZN5arrow7dataset19InsertImplicitCastsERKNS0_10ExpressionERKNS_6SchemaE + 36
      9 arrow.so 0x0000000104134e04 Z32dataset_ScannerBuilderFilterRKNSt3_110shared_ptrIN5arrow7dataset14ScannerBuilderEEERKNS0_INS2_10ExpressionEEE + 52
      10 arrow.so 0x00000001040f35b7 arrow_dataset_ScannerBuilder_Filter + 135
      11 libR.dylib 0x00000001001f375b R_doDotCall + 955
      12 libR.dylib 0x000000010023d46a bcEval + 99306
      13 libR.dylib 0x000000010022494d Rf_eval + 445
      14 libR.dylib 0x0000000100243209 R_execClosure + 2153
      15 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      16 libR.dylib 0x0000000100224e7d Rf_eval + 1773
      17 libR.dylib 0x0000000100245880 do_begin + 432
      18 libR.dylib 0x0000000100224b40 Rf_eval + 944
      19 libR.dylib 0x0000000100243209 R_execClosure + 2153
      20 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      21 libR.dylib 0x000000010022bd71 bcEval + 27889
      22 libR.dylib 0x000000010022494d Rf_eval + 445
      23 libR.dylib 0x0000000100243209 R_execClosure + 2153
      24 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      25 libR.dylib 0x0000000100287675 dispatchMethod + 757
      26 libR.dylib 0x0000000100287332 Rf_usemethod + 738
      27 libR.dylib 0x0000000100287926 do_usemethod + 646
      28 libR.dylib 0x000000010022c369 bcEval + 29417
      29 libR.dylib 0x000000010022494d Rf_eval + 445
      30 libR.dylib 0x0000000100243209 R_execClosure + 2153
      31 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      32 libR.dylib 0x0000000100224e7d Rf_eval + 1773
      33 libR.dylib 0x0000000100243209 R_execClosure + 2153
      34 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      35 libR.dylib 0x000000010022bd71 bcEval + 27889
      36 libR.dylib 0x000000010022494d Rf_eval + 445
      37 libR.dylib 0x00000001002418c3 forcePromise + 179
      38 libR.dylib 0x0000000100224c30 Rf_eval + 1184
      39 libR.dylib 0x0000000100247241 do_withVisible + 49
      40 libR.dylib 0x0000000100286603 do_internal + 339
      41 libR.dylib 0x000000010022c369 bcEval + 29417
      42 libR.dylib 0x000000010022494d Rf_eval + 445
      43 libR.dylib 0x0000000100243209 R_execClosure + 2153
      44 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      45 libR.dylib 0x000000010022bd71 bcEval + 27889
      46 libR.dylib 0x000000010022494d Rf_eval + 445
      47 libR.dylib 0x0000000100243209 R_execClosure + 2153
      48 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      49 libR.dylib 0x0000000100224e7d Rf_eval + 1773
      50 libR.dylib 0x0000000100243209 R_execClosure + 2153
      51 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      52 libR.dylib 0x0000000100224e7d Rf_eval + 1773
      53 libR.dylib 0x0000000100246bc6 do_eval + 646
      54 libR.dylib 0x000000010022c186 bcEval + 28934
      55 libR.dylib 0x000000010022494d Rf_eval + 445
      56 libR.dylib 0x0000000100243209 R_execClosure + 2153
      57 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      58 libR.dylib 0x000000010022bd71 bcEval + 27889
      59 libR.dylib 0x000000010022494d Rf_eval + 445
      60 libR.dylib 0x00000001002418c3 forcePromise + 179
      61 libR.dylib 0x0000000100224c30 Rf_eval + 1184
      62 libR.dylib 0x0000000100247241 do_withVisible + 49
      63 libR.dylib 0x0000000100286603 do_internal + 339
      64 libR.dylib 0x000000010022c369 bcEval + 29417
      65 libR.dylib 0x000000010022494d Rf_eval + 445
      66 libR.dylib 0x0000000100243209 R_execClosure + 2153
      67 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      68 libR.dylib 0x000000010022bd71 bcEval + 27889
      69 libR.dylib 0x000000010022494d Rf_eval + 445
      70 libR.dylib 0x0000000100243209 R_execClosure + 2153
      71 libR.dylib 0x000000010024210a Rf_applyClosure + 346
      72 libR.dylib 0x0000000100224e7d Rf_eval + 1773
      73 libR.dylib 0x000000010027506a Rf_ReplIteration + 794
      74 libR.dylib 0x000000010027658f run_Rmainloop + 207
      75 R 0x000000010016cf5b main + 27
      76 libdyld.dylib 0x00007fff7de493d5 start + 1
      Abort trap: 6

       

      Thanks to Neal Richardson for help with the above reprex and putting words to the issue - Neal says:

      "the 3 bugs are: (1) that should not crash; (2) R should translate Date in the filter expression to date32, just as it does for vectors; (3) the C++ layer should probably support that date64 to date32 cast (up for debate and technically not a bug, just not yet implemented I guess)".

       

      session_info():

      ─ Session info ──────────────────────────────────────────────────────────────────────────── setting value version R version 3.6.2 (2019-12-12) os macOS Mojave 10.14.6 system x86_64, darwin15.6.0 ui RStudio language (EN) collate en_CA.UTF-8 ctype en_CA.UTF-8 tz America/Vancouver date 2020-02-04 ─ Packages ──────────────────────────────────────────────────────────────────────────────── package * version date lib source arrow * 0.15.1.20200203 2020-02-03 [1] local assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.0) backports 1.1.5 2019-10-02 [1] CRAN (R 3.6.0) bit 1.1-15.1 2020-01-14 [1] CRAN (R 3.6.0) bit64 0.9-7 2017-05-08 [1] CRAN (R 3.6.0) bortles 0.0.0.9000 2019-11-12 [1] Github (adam-gruer/bortles@068a1b1) callr 3.4.1 2020-01-24 [1] CRAN (R 3.6.0) cli 2.0.1 2020-01-08 [1] CRAN (R 3.6.0) clisymbols 1.2.0 2017-05-21 [1] CRAN (R 3.6.0) crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.0) desc 1.2.0 2018-05-01 [1] CRAN (R 3.6.0) devtools * 2.2.1 2019-09-24 [1] CRAN (R 3.6.1) digest 0.6.23 2019-11-23 [1] CRAN (R 3.6.0) dplyr * 0.8.4 2020-01-31 [1] CRAN (R 3.6.0) ellipsis 0.3.0 2019-09-20 [1] CRAN (R 3.6.0) evaluate 0.14 2019-05-28 [1] CRAN (R 3.6.0) fansi 0.4.1 2020-01-08 [1] CRAN (R 3.6.0) fs 1.3.1 2019-05-06 [1] CRAN (R 3.6.0) glue 1.3.1 2019-03-12 [1] CRAN (R 3.6.0) here * 0.1 2017-05-28 [1] CRAN (R 3.5.0) htmltools 0.4.0 2019-10-04 [1] CRAN (R 3.6.0) knitr 1.27 2020-01-16 [1] CRAN (R 3.6.2) magrittr 1.5 2014-11-22 [1] CRAN (R 3.6.0) memoise 1.1.0 2017-04-21 [1] CRAN (R 3.6.0) packrat 0.5.0 2018-11-14 [1] CRAN (R 3.6.0) pillar 1.4.3 2019-12-20 [1] CRAN (R 3.6.0) pkgbuild 1.0.6 2019-10-09 [1] CRAN (R 3.6.0) pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 3.6.0) pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.6.0) praise 1.0.0 2015-08-11 [1] CRAN (R 3.5.0) prettyunits 1.1.1 2020-01-24 [1] CRAN (R 3.6.0) processx 3.4.1 2019-07-18 [1] CRAN (R 3.6.0) prompt 1.0.0 2020-01-13 [1] Github (gaborcsardi/prompt@b332c42) ps 1.3.0 2018-12-21 [1] CRAN (R 3.6.0) purrr 0.3.3 2019-10-18 [1] CRAN (R 3.6.0) R6 2.4.1 2019-11-12 [1] CRAN (R 3.6.0) Rcpp 1.0.3 2019-11-08 [1] CRAN (R 3.6.0) remotes 2.1.0 2019-06-24 [1] CRAN (R 3.6.0) reprex 0.3.0 2019-05-16 [1] CRAN (R 3.6.0) rlang 0.4.4 2020-01-28 [1] CRAN (R 3.6.0) rmarkdown 2.1 2020-01-20 [1] CRAN (R 3.6.0) rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.0) rstudioapi 0.10 2019-03-19 [1] CRAN (R 3.6.0) sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.0) testthat * 2.3.1 2019-12-01 [1] CRAN (R 3.6.0) tibble 2.1.3 2019-06-06 [1] CRAN (R 3.6.0) tictoc * 1.0 2014-06-17 [1] CRAN (R 3.6.0) tidyselect 1.0.0 2020-01-27 [1] CRAN (R 3.6.0) usethis * 1.5.1 2019-07-04 [1] CRAN (R 3.6.0) vctrs 0.2.2 2020-01-24 [1] CRAN (R 3.6.0) whisker 0.4 2019-08-28 [1] CRAN (R 3.6.0) withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.0) xfun 0.12 2020-01-13 [1] CRAN (R 3.6.0)

      Attachments

        Issue Links

          Activity

            People

              bkietz Ben Kietzman
              stephhazlitt Stephanie Hazlitt
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h