[ARROW-6230] [R] Reading in Parquet files are 20x slower than reading fst files in R

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.15.0
• Component/s: R
• Labels: parquet
• Environment: Windows 10 Pro and Ubuntu
• External issue URL: https://github.com/apache/arrow/issues/22617
• Assignee: Wes McKinney
• Reporter: Zhuo Jia Dai
• Created: 14/Aug/19 00:06
• Updated: 11/Jan/23 07:45
• Resolved: 15/Aug/19 16:18

Description

Problem

Loading any of the data I mentioned below is 20x slower than the fst format in R.

 

How to get the data

https://loanperformancedata.fanniemae.com/lppub/index.html

Register and download any of these. I can't provide the data to you, and I think it's best you register.

 

 

Code

```r
path = "data/Performance_2016Q4.txt"

library(data.table)
library(arrow)

a = data.table::fread(path, header = FALSE)

fst::write_fst(a, "data/a.fst")

arrow::write_parquet(a, "data/a.parquet")

rm(a); gc()

#read in test
system.time(a <- fst::read_fst("data/a.fst")) # 4.61 seconds

rm(a); gc()

read in test
system.time(a <- arrow::read_parquet("data/a.parquet") # 99.19 seconds
```

Issue Links

• relates to: ARROW-6060 [Python] too large memory cost using pyarrow.parquet.read_table with use_threads=True