[ORC-469] Add EncodedStringVectorBatch to expose string dictionary in VectorBatch

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.5, 1.6.0
• Component/s: Reader
• Assignee: Yurui Zhou
• Reporter: Yurui Zhou
• Created: 13/Feb/19 08:16
• Updated: 25/May/22 04:14
• Resolved: 03/Apr/19 05:11

Description

Propose to add EncodedStringVectorBatch to expose string dictionary in VectorBatch. Exposing string dictionary will benefits as bellow:

• Enable computation over encoded data. By exposing dictionary in vector batch we would be able to implement filter operator in a more efficient way. In our POC, by enabling encoded data based filter, we achieved 8% E2E perf improvement on tpch q1
• Make data serialization more efficient. Currently when serializing Orc Vectorbatch, we have to copy all the strings in a vector even though the string data is already dictionary encoded. Exposing String Dictionary will enable vector batch serializer to remove unnecessary string memcpy, which will greatly improve serialization efficiency.

Issue Links

• causes: ORC-590 Crash in orc::RleDecoderV2::readByte
• links to: GitHub Pull Request #367