Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
Description
Hi developers
Saving Arrow::Table in CSV format may be slow.
Ad hoc benchmarks...
require 'arrow' require 'csv' require 'gr/plot' t = Arrow::Table.load('some_nice.tsv', format: :csv, delimiter: "\t".ord) n = 1.step(1000, 100).to_a arrow_save_times = [] csv_save_times = [] n.each do |i| t2 = t.slice(0, i) start = Time.now t2.save('test.csv') arrow_save_times << p(Time.now - start) t2 = t.raw_records start = Time.now CSV.open('test2.csv', 'w') do |csv| t2.each do |r| csv << r end end csv_save_times << p(Time.now - start) end GR.stem([n, arrow_save_times], [n, csv_save_times], labels: ["arrow", "CSV"], xlabel: "lines", ylabel: "time", location: 2) GR.savefig("arrow.png") gets
Attachments
Attachments
Issue Links
- links to