[ARROW-6481] [Python][C++] Bad performance of read_csv() with column_types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.1
• Fix Version/s: 0.15.0
• Component/s: C++, Python
• Labels: pull-request-available
• Environment: ubuntu xenial, python2.7
• External issue URL: https://github.com/apache/arrow/issues/22850
• Assignee: Antoine Pitrou
• Reporter: Bogdan Klichuk
• Created: 07/Sep/19 23:52
• Updated: 11/Jan/23 07:47
• Resolved: 10/Sep/19 14:41

Description

Case: Dataset wit 20k columns. Amount of rows can be 0.

pyarrow.csv.read_csv('20k_cols.csv') works rather fine if no convert_options provided.

Took 150ms.

Now I call read_csv() with column types mapping that marks 2000 out of these columns as string.

pyarrow.csv.read_csv('20k_cols.csv', convert_options=pyarrow.csv.ConvertOptions(column_types={'K%d' % i: pyarrow.string() for i in range(2000)}))

(K1..K19999 are column names in attached dataset).

My task globally is to read everything as string, avoid any inferring.

This takes several minutes, consumes around 4GB memory.

This doesn't look sane at all.

Issue Links

• links to: GitHub Pull Request #5334