[ARROW-6786] [C++] arrow-dataset-file-parquet-test is slow

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: C++
• External issue URL: https://github.com/apache/arrow/issues/23122
• Assignee: Unassigned
• Reporter: Antoine Pitrou
• Created: 03/Oct/19 19:19
• Updated: 11/Jan/23 07:49
• Resolved: 29/Aug/22 14:14

Description

It takes 15 seconds in debug mode (probably more with ASAN / UBSAN /etc.) to run 2 tests that simply iterated through a generated in-memory dataset:

$ ./build-test/debug/arrow-dataset-file-parquet-test 
Running main() from /home/conda/feedstock_root/build_artifacts/gtest_1551008230529/work/googletest/src/gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from TestParquetFileFormat
[ RUN      ] TestParquetFileFormat.ScanRecordBatchReader
[       OK ] TestParquetFileFormat.ScanRecordBatchReader (7338 ms)
[ RUN      ] TestParquetFileFormat.Inspect
[       OK ] TestParquetFileFormat.Inspect (6222 ms)
[----------] 2 tests from TestParquetFileFormat (13560 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (13560 ms total)
[  PASSED  ] 2 tests.

Unless it is stressing something in particular, the number of repetitions or the batch size can probably be reduced dramatically.

Issue Links

• is superceded by: ARROW-6951 [C++][Dataset] Ensure column projection is passed to ParquetDataFragment