[ARROW-7252] [Rust] [Parquet] Reading UTF-8/JSON/ENUM field results in a lot of vec allocation

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Invalid
• Fix Version/s: 5.0.0
• Component/s: Rust
• External issue URL: https://github.com/apache/arrow/issues/23545
• Assignee: Unassigned
• Reporter: Wong Shek Hei
• Created: 25/Nov/19 15:29
• Updated: 11/Jan/23 07:52
• Resolved: 26/Apr/21 11:24

Description

While reading a very large parquet file with basically all string fields was very slow(430MB gzipped), after profiling with osx instruments, I noticed that a lot of time is spent in "convert_byte_array", in particular, "reserving" and allocating Vec::with_capacity, which is done before String::from_utf8_unchecked.

It seems like using String as the underlying storage is causing this(String uses Vec<u8> for its underlying storage), this also requires copying from slice to vec.

"Field::Str<String>" is a pub enum so I am not sure how "refactorable" is the String part, for example, converting it into a &str(we can perhaps then defer the conversion from &[u8] to Vec<u8> until the user really needs a String)

But of course, changing it to &str can result in quite a bit of interface changes... So I am wondering if there are already some plans or solution on the way to improve the handling of the "Field::Str" case?