Major Description
This is found by drorke that orc::RleDecoderV2::nextDirect takes the majority of the time when scanning bigint columns. I reproduce the issue by using the orc-scan tool to read the random bigint columns of a TPCH lineitem table. In the attached frame graph, 91.89% of the time is spent in orc::RleDecoderV2::nextDirect. Only a small portion of it is used in snappy decompression.
Note that orc::RleDecoderV2::nextDirect is also used in other column types, e.g. dictionary encoded string columns. So improving it can boost performance in many scenarios.
We should consider unrolling the loop in orc::RleDecoderV2::readLongs. There is already a TODO: https://github.com/apache/orc/blob/93af6b076c210b0c3b77e5af3d6fbef1bd1150a1/c%2B%2B/src/RLEv2.hh#L186
csringhofer also points out that we can borrow some ideas done in Impala for bit unpacking: https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/util/bit-packing.inline.h#L60
Attachments
Attachments
Issue Links
- links to