[ARROW-7474] [Ruby] Save CSV files faster

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Ruby
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/23741
• Assignee: Kouhei Sutou
• Reporter: kojix2
• Created: 27/Dec/19 06:47
• Updated: 11/Jan/23 07:54
• Resolved: 31/Dec/19 01:03

Description

Hi developers

Saving Arrow::Table in CSV format may be slow.

Ad hoc benchmarks...

 

 
require 'arrow'
require 'csv'
require 'gr/plot'
t = Arrow::Table.load('some_nice.tsv', format: :csv, delimiter: "\t".ord)
n = 1.step(1000, 100).to_a
arrow_save_times = []
csv_save_times = []
n.each do |i|
 t2 = t.slice(0, i)
start = Time.now
 t2.save('test.csv')
 arrow_save_times << p(Time.now - start)
t2 = t.raw_records
start = Time.now
 CSV.open('test2.csv', 'w') do |csv|
 t2.each do |r|
 csv << r
 end
 end
 csv_save_times << p(Time.now - start)
end
GR.stem([n, arrow_save_times], [n, csv_save_times],
 labels: ["arrow", "CSV"], xlabel: "lines", ylabel: "time", location: 2)
GR.savefig("arrow.png")
gets

 

 

Issue Links

• links to: GitHub Pull Request #6106